search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.39k stars 3.39k forks source link

Docker driver doesn't work with warning "no logs are available with the 'loki' log driver" #1368

Closed rndmit closed 4 years ago

rndmit commented 4 years ago

Describe the bug Docker driver configured as written in docs but after running docker-compose up docker shows warning. And there's also no logs in Loki.

To Reproduce Steps to reproduce the behavior:

  1. Started Loki (master)
  2. Installed and enabled Docker-driver via docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions command
  3. Added logging section to docker-compose file as said in docs
  4. Run docker-compose up
  5. See the warning: app_1 | WARNING: no logs are available with the 'loki' log driver

Expected behavior Logs should fall into Loki but they don't.

Environment:

Screenshots, Promtail config, or terminal output docker-compose.yml

version: '3.7'

services:
  loki:
    image: grafana/loki:master
    container_name: loki
    command: -config.file=/etc/loki/local-config.yaml
  app:
    build:
      context: .
      dockerfile: Dockerfile
    logging:
      driver: loki
      options:
        loki-url: "http://loki:3100/loki/api/v1/push"
cyriltovena commented 4 years ago

This is because the driver runs with host network, I need to try bridge and make it defaults.

Edit: not sure actually

inliquid commented 4 years ago

Same weird behavior in my case.

dmitriyminer commented 4 years ago

@inliquid, @scobcov Try this one:

version: '3.7'

services:
  grafana:
    image: grafana/grafana:master
    ports:
      - "3000:3000"
    environment:
      GF_EXPLORE_ENABLED: "true"

  loki:
    image: grafana/loki:master
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  app:
    image: alpine:latest
    logging:
      driver: loki
      options:
        loki-url: "http://host.docker.internal:3100/loki/api/v1/push"
    entrypoint: ping host.docker.internal
rndmit commented 4 years ago

@dmitriyminer well, it works but looks like a kludge. I think it's a really bad idea to expose Loki outside. However, thanks. I'll use it as a temporary solution and restrict access with a firewall. Btw, I'm still waiting for a solution from devs.

cyriltovena commented 4 years ago

We don’t have a solution because we are restrained by how docker plugins work which is either bridge or host network, currently it configure as host. I’m really busy to try out bridge which should solve most use cases except custom network. Wanna try bridge here is an image:

grafana/loki-docker-driver:latest-bridge

tomdsmartdata commented 4 years ago

@cyriltovena, I'm encountering a similar as @scobcov and am interested in trying your bridge image. Where can I pull it from?

cyriltovena commented 4 years ago

grafana/loki-docker-driver:latest-bridge

rhbroberg commented 4 years ago

As of 2.2.0.3 (42716) DockerDesktop on the mac, or 19.03.5 on linux, neither of these approaches works for me: both of them result in the same error message from 'docker-compose logs':

Attaching to docker-loki_app_1, docker-loki_grafana_1, docker-loki_loki_1 app_1 | WARNING: no logs are available with the 'loki' log driver

I'm using the exact docker-compose file as posted above, and I've tried it with the grafana/loki-docker-driver:latest-bridge plugin installed. Any suggestions?

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

moeenz commented 4 years ago

I'm experiencing the same thing on Mac and Ubuntu. Tried with both :latest and :latest-bridge. Exposing Loki port or not, the problem persists.

cyriltovena commented 4 years ago

Hey Everyone,

For those who have the issue related to no logs are available this is in fact on purpose from the docker-compose tooling.

see https://github.com/docker/compose/blob/master/compose/container.py#L196

Basically no log from docker-compose if you use a plugin, not sure why. But docker logs should work still, if you want to use that.

andoks commented 4 years ago

I too have experienced this issue. I have worked around it by binding port 3100 of the Loki container to the hosts' port (127.0.0.1:3100), but I expect that when I'm going to run more than one compose-project on the same host, each with their own Loki container inside, this will break down and I will have to change the compose-file.

I would love for this issue to be reopened. A solution to the issue I would really like, is if the loki logging driver got an option for specifying that the push-url is inside the source docker bridge network (or something more powerful and flexible if there are more use-cases to cover)

cyriltovena commented 4 years ago

@andoks what you asking is not really related to this issue. But if you want to try the bridge network be my guest here is the image grafana/loki-docker-driver:latest-bridge, so far no one has reported that it fixed the network issue.

From my understanding this is a docker limitation.

andoks commented 4 years ago

what you asking is not really related to this issue

@cyriltovena in that case I am sorry.

I thought the original issue as described was "I am not able to send logs to the container running inside the compose setup using the docker loki logging driver".

And that dmitriyminer in his post provided a work-around for this by suggesting binding lokis' API port to the host to make it available to the logging driver.

But if you want to try the bridge network be my guest here is the image grafana/loki-docker-driver:latest-bridge, so far no one has reported that it fixed the network issue.

From my understanding this is a docker limitation.

I tested it in place of grafana/loki-docker-driver:latest using loki 1.5.0 in a container. It did not solve my issue with wanting to log to the loki container running inside the compose setup without binding the loki API port to a host port. Actually, the work-around (binding the loki-container api port to the host) did not work with the latest-bridge. Should it have worked?

cyriltovena commented 4 years ago

Not if you're using the default bridge.

see https://docs.docker.com/network/bridge/#differences-between-user-defined-bridges-and-the-default-bridge

inliquid commented 4 years ago

not really related to this issue

It's absolutely related. Many ppl asked about this in Slack as well. The problem is that Loki Docker driver should recognize docker host names instead of just one single 127.0.0.1.

cyriltovena commented 4 years ago

not really related to this issue

It's absolutely related. Many ppl asked about this in Slack as well. The problem is that Loki Docker driver should recognize docker host names instead of just one single 127.0.0.1.

It has nothing to do with Loki, I've answered it many times, this is a docker plugin restrictions, we can only set the network to either host or bridge.

Please if I'm wrong show me in the doc where I can allow the driver to access docker network.

cyriltovena commented 4 years ago

see image

andoks commented 4 years ago

@cyriltovena: I gave the latest-bridge another go, but I was still unable to solve my problem with it (and unable to actually get data into loki using the plugin, but that might just be something I did wrong).

After reading up a bit, I think I understood the bridge feature better: that instead of binding to the host network, it binds to the built-in docker default network called "bridge" (the description is kinda terse. This was my interpretation of it at least, please correct me if I am wrong).

This means that the loki instance I have currently running in my docker-compose setup still is unavailable for the loki driver plugin in "bridge" mode. I tried experimenting with putting the loki container inside the "bridge" network, and learned some of the pains of using this network: no built-in service dns, not possible to be connected to a user-defined docker network at the same time as you are connected to the built-in "bridge" network etc.

After discovering the two issues above, this was enough for me to conclude that binding the loki-containers' port to the docker host, and pushing logs using the loki driver plugin as the work-around described above, will be far less of a hassle. The issue with this of course, is that it will not be possible to run multiple systems from the same compose-file without changing that port. It is not perfect, but it is something I got working, and it is relatively obvious (to describe, and to reason about).

Thanks for your help trying to find a solution to this.

PS: is this a TBD repo?

cyriltovena commented 4 years ago

Thanks for the update, I think docker never really intended to allow logging driver to connect to interval network and that’s why it’s like this.

If someone wants to push this further we could open an issue on moby/docker side.

And no that repo is not used, we will keep the code of the driver in the main repository.

tam481 commented 4 years ago

Hello all, I'm confused. Are we saying that the docker loki driver does not work if the containers are started using docker-compose because of some limitation?!

I have loki running behind traefik and the driver configured to push docker logs to it via docker-compose.yml for the services I am running. None of them seem to be logging anything to loki.

Loki returns ready when I visit (https://loki.domain.tld/ready) but nothing is actually being logged.

I tried the "grafana/loki-docker-driver:latest-bridge" to no avail

denisgolius commented 4 years ago

what you asking is not really related to this issue

@cyriltovena in that case I am sorry.

I thought the original issue as described was "I am not able to send logs to the container running inside the compose setup using the docker loki logging driver".

And that dmitriyminer in his post provided a work-around for this by suggesting binding lokis' API port to the host to make it available to the logging driver.

But if you want to try the bridge network be my guest here is the image grafana/loki-docker-driver:latest-bridge, so far no one has reported that it fixed the network issue. From my understanding this is a docker limitation.

I tested it in place of grafana/loki-docker-driver:latest using loki 1.5.0 in a container. It did not solve my issue with wanting to log to the loki container running inside the compose setup without binding the loki API port to a host port. Actually, the work-around (binding the loki-container api port to the host) did not work with the latest-bridge. Should it have worked?

docker pull grafana/loki-docker-driver:latest

Using default tag: latest
Error response from daemon: pull access denied for grafana/loki-docker-driver, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Or with latest-bridge

docker pull grafana/loki-docker-driver:latest-bridge
Error response from daemon: pull access denied for grafana/loki-docker-driver, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
andoks commented 4 years ago

@tam481: no, the docker log loki driver works fine, it is just that since it is sitting on the host network, it does not have automatic access to the loki container running within a docker-compose project. The way I made this work, was to bind the loki containers log input port to host, and then the docker log loki driver plugin has access to write logs to loki over the host port

@denisgolius: I am pretty sure you see those errors due to pulling (docker pull) the images instead of installing them using the plugin command.

tam481 commented 4 years ago

hi @andoks I'm running Loki behind Traefik v2 reverse-proxy. I can log to it just fine from other devices using promtail. However I cannot get docker to ship its logs using the driver. Port 443 is bound to the IP address of the device so when you run docker ps you can clearly see that port 443 is bound to it correctly but I still can't get docker logs shipped to Loki from the local docker! I don't have docker running elsewhere so I can't test it from a remote device.

tam481 commented 4 years ago

Hello all I hope this helps someone.

I resorted to assigning a static address to the Loki container in Docker e.g. 172.20.0.10 and configured the docker-compose projects to log to Loki internally via IP address

 &loki
  logging:
    driver: loki
    options:
      loki-url: "http://172.20.0.10:3100/loki/api/v1/push"

Logging is working perfectly now. Externally, logs are going via Traefik on port 443 reverse-proxied to 3100

cyriltovena commented 4 years ago

Sounds like a good work around well done.

clouedoc commented 3 years ago

@tam481 Thank you so much!

I read this issue a lot but not carefully, so I'm going to lay down steps for people who are encountering the issue but are too lazy to read.

Steps: 1) create a virtual network 2) assign a static ip to your loki service

Here is a working docker-compose.yml:

version: "3.8"

networks:
  vpcbr: # virtual network name
    driver: bridge
    ipam:
     config: # here, we define our ip space
       - subnet: 10.5.0.0/16
         gateway: 10.5.0.1

# this is a YAML anchor to avoid repetition
x-logging: &logging
  logging:
    driver: loki
    options: # note: the ip is the ip we will assign later to loki in this file
      loki-url: "http://10.5.0.2:3100/loki/api/v1/push"

services:
  # gitea is a demo service from which we want to extract logs
  gitea:
    <<: *logging
    image: gitea/gitea:1.13.0
    container_name: gitea
    # here I removed informations irrelevant to the issue

  # the loki instance
  loki:
    image: grafana/loki:master
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      vpcbr: # this is the place where we assign the static ipv4 address
        ipv4_address: 10.5.0.2

Please try these steps before mindlessly wondering on Google ! And note to myself: always fully read GitHub issues before resorting to Google binge-searching.

It works: image

fmoledina commented 3 years ago

@clouedoc, this looks really good. Are you using the grafana/loki-docker-driver:latest or grafana/loki-docker-driver:latest-bridge plugin?

fmoledina commented 3 years ago

I got it working using grafana/loki-docker-driver:2.1.0 and grafana/loki-docker-driver:2.1.0. Thanks!

chintanp commented 3 years ago

@clouedoc Thank you for posting in details your docker-compose config for making this work. Can you also post your grafana service config? I am using it in the same compose file, and it gets assigned an IP address from the range defined in the network, but I am not sure how to access it from outside. As in how does I re-route my nginx request to the grafana container?

PS: The grafana instance is showing logs collected by loki from the docker-compose stack. It would be nice to be able to access Grafana over internet. I am using an Ubuntu EC2 machine for my docker-compose stack, and was able to get to Grafana using remote desktop. loki

clouedoc commented 3 years ago

@clouedoc Thank you for posting in details your docker-compose config for making this work. Can you also post your grafana service config? I am using it in the same compose file, and it gets assigned an IP address from the range defined in the network, but I am not sure how to access it from outside. As in how does I re-route my nginx request to the grafana container?

PS: The grafana instance is showing logs collected by loki from the docker-compose stack. It would be nice to be able to access Grafana over internet. I am using an Ubuntu EC2 machine for my docker-compose stack, and was able to get to Grafana using remote desktop. loki

Hi, I don't have any specific config for grafana. The ip address should not be a random one from the range, but exactly the one that is specified in the docker-compose.yml (here, 10.5.0.2).

To access Grafana from outside, you need to add a reverse proxy entry in your nginx config that redirects requests from the outside to 10.5.0.2. I never used nginx so I don't know how you should achieve that. It's a trivial task though, so you should be able to do it ;).

chintanp commented 3 years ago

@clouedoc Thanks. I was able to figure it out. This docker forum post describes my situation and resolution. My docker-compose file if anyone else is interested. My final question is: do we need to specify static IP addresses for all containers (as otherwise if we create one of the non-assigned containers in the stack before, it can get auto-assigned an IP address that was assigned to another container). And how can we access the Loki instance over the internet? I want to send logs to it from the outside.

clouedoc commented 3 years ago

@clouedoc Thanks. I was able to figure it out. This docker forum post describes my situation and resolution. My docker-compose file if anyone else is interested. My final question is: do we need to specify static IP addresses for all containers (as otherwise if we create one of the non-assigned containers in the stack before, it can get auto-assigned an IP address that was assigned to another container). And how can we access the Loki instance over the internet? I want to send logs to it from the outside.

To access the Loki instance from the internet, you can setup a Nginx reverse proxy pointing to it. I don't think that Docker will assign the same IP to two containers, but I'm not sure of this. The best way to know is to try it ! Or spend hours to search for an answer to an unexisting question.

chintanp commented 3 years ago

Docker was in fact assigning addresses that were assigned. It might not do that in DHCP mode, but partial manual addressing may not work. I had to assign manual addresses to all. Thanks.

chiqui3d commented 3 years ago

@clouedoc I really appreciate the solution, and it works great, but there is something I don't understand and i hope someone can help me understand it. In all the services I have put the logging driver of Loki and it works like a charm, besides some services are not connected to the network created (vpcbr). So how Loki logging can access that IP and I can't? I mean if I access a container that doesn't share the network and where the driver is connected, I can't do CURL to IP 10.5.0.2:3100/loki/api/v1/label/foo/values as in your example.

clouedoc commented 3 years ago

@chiqui3d this post is too old for me to remember the specifics, but I believe that you can't access this IP because it's internal to Docker's network routing. It also can be that you're using Windows or MacOS and this ip is confined inside your Docker VM (because it will run on a transparent VM when not using Linux).

michaelKaefer commented 3 years ago

On Ubuntu host.docker.internal has to be configured manually, see my example and the link in the comment. My example works when using Traefik:

  nginx:
    # ...
    logging:
      driver: loki
      options:
        loki-url: "http://host.docker.internal:3100/loki/api/v1/push"
    extra_hosts:
      # See https://stackoverflow.com/questions/48546124/what-is-linux-equivalent-of-host-docker-internal/67158212#67158212
      # Used for sending logs to Loki
      - "host.docker.internal:host-gateway"
KES777 commented 1 year ago

@cyriltovena: this post should referenced from documentation https://grafana.com/docs/loki/latest/clients/docker-driver/

KES777 commented 1 year ago

Seems found shorter solution: just do not use service name at loki-url, use static IP That is all.

Probably compose still looks scary, but actually it is not.

networks:
  office-net:
    name: office-net
    driver: bridge
    ipam:
      config:
        - subnet: "172.22.22.0/24"

services:
  loki:
    image: grafana/loki:latest
    command: [
      "-config.file=/etc/loki/local-config.yml",
      "-log.level=debug",
    ]
    volumes:
      - ${STACK_ROOT}/loki-config.yml:/etc/loki/local-config.yml
      - loki-data:/loki
    logging:
      driver: loki
      options:
        loki-url: "http://172.22.22.221:3100/loki/api/v1/push"
        loki-external-labels: "container_name={{.Name}},job=dockerlogs"
    networks:
      office-net:
        ipv4_address: "172.22.22.221"

  promtail:
    depends_on:
      - loki
    image: grafana/promtail:latest
    command: -config.file=/etc/promtail/config.yml
    volumes:
      - /var/log:/var/log
      - ${STACK_ROOT}/promtail-config.yml:/etc/promtail/config.yml
    logging:
      driver: loki
      options:
        loki-url: "http://172.22.22.221:3100/loki/api/v1/push"
        loki-external-labels: "container_name={{.Name}},job=dockerlogs"
    networks:
      office-net:

Why that error happened? Because plugin not on the network and have no access to service's name, but because logging driver is on the bridge, the network is visible to it, so loki is accessible by static IP address.

This should be documented and highlighted by bold red font!