search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.89k stars 3.45k forks source link

deploy loki cluster on swarm : transport error #6648

Open dginhoux opened 2 years ago

dginhoux commented 2 years ago

Hi,

I use a swarm cluster and i'm trying to deploy the provided example docker-compose : https://github.com/grafana/loki/tree/main/production/docker on a swarm cluster instead of docker-compose tool.

Describe the bug both loki services (1, 2 and 3) are looping this kind of logs : loki_loki-2.1.e67fhahggiw7@pc-dg18-u | level=error ts=2022-07-08T22:45:59.379302282Z caller=frontend_scheduler_worker.go:231 msg="error contacting scheduler" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 10.0.0.23:9095: i/o timeout\"" addr=10.0.0.23:9095

To Reproduce Steps to reproduce the behavior:

  1. init a swarm cluster, can be made with just one node/computer "docker swarm init"
  2. git clone the repo
  3. go to production/docker
  4. "upgrade" 2.5.0 do 2.6.0 in docker-compose-ha-memberlist.yaml
  5. mkdir -m 777 loki
  6. sudo docker stack deploy -c docker-compose-ha-memberlist.yaml loki
  7. sudo docker service logs -f loki_XXXXXX where XXXX is the service name

Environment: get a fresh linux, with docker installed and no firewall, selinux/apparmor disabled

hhromic commented 2 years ago

I found this same issue some time ago since Loki 2.4.x: https://github.com/grafana/loki/issues/4699#issuecomment-971526868

The issue seems to be when deploying Loki within the Swarm ingress network (the default service deploy mode).

After some investigation and discussion, managed to make it work in Swarm mode: https://github.com/grafana/loki/issues/4699#issuecomment-1027866002

In summary, for Loki 2.6.x you have to force the component addresses to use 127.0.0.1 to override the default interface-name options (which often start with eth0, causing the conflict with the ingress network):

version: '3.8'

services:
  loki:
    image: grafana/loki:2.6.1
    command: >-
      -config.file=/etc/loki/local-config.yaml
      -boltdb.shipper.compactor.ring.instance-addr=127.0.0.1
      -distributor.ring.instance-addr=127.0.0.1
      -frontend.instance-addr=127.0.0.1
      -index-gateway.ring.instance-addr=127.0.0.1
      -ingester.lifecycler.addr=127.0.0.1
      -query-scheduler.ring.instance-addr=127.0.0.1
      -ruler.ring.instance-addr=127.0.0.1
    ports:
      - 3100:3100
    volumes:
      - data:/loki

volumes:
  data: {}

For Loki 2.5.x, the above should work as well. At least from my testing, only this was sufficient in Loki 2.5.x:

-frontend.instance-addr=127.0.0.1

Of course, the above will not work when replicating the service beyond 1 replica.

dginhoux commented 2 years ago

Hi,

Thank for your test and answer. Scaling is totally impossible... Maybe one service per node and again.... I'm not sure it can work.

hhromic commented 2 years ago

As an alternative, you can also deploy without the Swarm ingress network and then no need for the CLI args:

version: '3.8'

services:
  loki:
    image: grafana/loki:2.6.1
    ports:
      - target: 3100
        published: 3100
        protocol: tcp
        mode: host
    volumes:
      - data:/loki

volumes:
  data: {}

However, without the ingress network, it will only listen on the deployed node. In this case, you can use Traefik (easiest) or any reverse proxy in front of Loki to provide distributed ingress capabilities.

I have never deployed Loki with more than one replica, so I don't really know how to configure it to scale.

hhromic commented 2 years ago

And here is an example using Traefik. Note that the loki service does not expose a port itself, therefore not participating in the Swarm ingress network.

version: '3.8'

services:
  loki:
    image: grafana/loki:2.6.1
    volumes:
      - data:/loki
    deploy:
      endpoint_mode: dnsrr
      mode: replicated
      replicas: 1
      labels:
        traefik.loki: 'true'
        traefik.http.routers.loki.rule: PathPrefix(`/loki/`)
        traefik.http.services.loki.loadbalancer.server.port: 3100
  traefik:
    image: traefik:v2.8.1
    environment:
      TRAEFIK_API_INSECURE: 'true'
      TRAEFIK_ENTRYPOINTS_DEFAULT_ADDRESS: ':5555'
      TRAEFIK_GLOBAL_CHECKNEWVERSION: 'false'
      TRAEFIK_GLOBAL_SENDANONYMOUSUSAGE: 'false'
      TRAEFIK_PROVIDERS_DOCKER_CONSTRAINTS: Label(`traefik.loki`, `true`)
      TRAEFIK_PROVIDERS_DOCKER_ENDPOINT: 'unix:///var/run/docker.sock'
      TRAEFIK_PROVIDERS_DOCKER_SWARMMODE: 'true'
    ports:
      - mode: host
        published: 5555
        protocol: tcp
        target: 5555
      - mode: host
        published: 8080
        protocol: tcp
        target: 8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      endpoint_mode: dnsrr
      mode: global    
      placement:
        constraints:
          - node.role == manager

volumes:
  data: {}

In the above example, Loki will respond on port 5555 from any MANAGER node of the Swarm cluster thanks to Traefik. I also enabled the Traefik dashboard on port 8080 for inspection of what is going on.

It is possible to deploy Traefik in all nodes (regardless of manager role) by using the docker-socket-proxy container as detailed for example here: https://www.rockyourcode.com/traefik-2-docker-swarm-setup-with-docker-socket-proxy-and-more/

A limitation of the above setup is that only Loki endpoints with prefix /loki are accessible. This is all the endpoints you will proabably use for Loki, i.e. from Grafana or from log collectors.

dginhoux commented 2 years ago

Hi,

Great idea to use a frontend... but what about scale up/down ? and the most important for me, split services read/write, compactor, indexer, etc...?

I use a dedicated swarm cluster for victoria metrics and i'm building for the same thing with loki.

In my "dreams", i have on scalable haproxy/nginx instance for incoming logs, and one other for output. And yes, everything scale manualy here.... not a problem for me.

I agree with you, i've built my own socat image for share docker socket instead mounting docker.sock to everyone need it, i use a dedicated overlay network.

hhromic commented 2 years ago

but what about scale up/down ? and the most important for me, split services read/write, compactor, indexer, etc...?

As I said before, unfortunately we haven't need to use Loki in distributed mode, so I have no experience with setting it up with multiple replicas. For now I can't really help much in that regard :(

Recent versions of Loki introduced Simple Scalable Mode, which seems to be an interesting mode to explore.

dginhoux commented 2 years ago

My colleague report me the same error with this simple mode.

He does not use scalable services, juste 2 services, one Read and one Write.

I'll follow here the used compose file on the evening.

dginhoux commented 2 years ago

Hi,

This is my latest work on this stack : docker-compose.yml.txt loki.yml.txt

Each write and read service, run and start fine, bu the ring doesn't initialize.... Clients (like grafana or promtail) get this error : "server returned HTTP status 500 Internal Server Error: at least 2 live replicas required, could only find 1"

Logs from read service : 

loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:44.70447276Z caller=scheduler.go:682 msg="this scheduler is in the ReplicationSet, will now accept requests."
loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:44.70574119Z caller=worker.go:209 msg="adding connection" addr=10.0.3.30:9095
loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:46.702682531Z caller=compactor.go:386 msg="this instance has been chosen to run the compactor, starting compactor"
loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:46.702786393Z caller=compactor.go:413 msg="waiting 10m0s for ring to stay stable and previous compactions to finish before starting compactor"
loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | ts=2022-07-26T18:24:50.524455522Z caller=memberlist_logger.go:74 level=info msg="Suspect 6791bdb44422-5ef1b790 has failed, no acks received"
loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:51.705043625Z caller=frontend_scheduler_worker.go:101 msg="adding connection to scheduler" addr=10.0.3.30:9095
loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | ts=2022-07-26T18:25:05.521981582Z caller=memberlist_logger.go:74 level=info msg="Suspect 6791bdb44422-5ef1b790 has failed, no acks received"
loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | ts=2022-07-26T18:25:10.524965803Z caller=memberlist_logger.go:74 level=info msg="Marking 6791bdb44422-5ef1b790 as failed, suspect timeout reached (0 peer confirmations)"
loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | ts=2022-07-26T18:25:25.521840094Z caller=memberlist_logger.go:74 level=info msg="Suspect 6791bdb44422-5ef1b790 has failed, no acks received"
loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | level=info ts=2022-07-26T18:29:40.498393288Z caller=table_manager.go:213 msg="syncing tables"
loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | level=info ts=2022-07-26T18:29:40.498456497Z caller=table_manager.go:252 msg="query readiness setup completed" duration=1.983µs distinct_users_len=0

loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | level=info ts=2022-07-26T18:34:40.498655335Z caller=table_manager.go:213 msg="syncing tables"
loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | level=info ts=2022-07-26T18:34:40.498703012Z caller=table_manager.go:252 msg="query readiness setup completed" duration=1.652µs distinct_users_len=0
loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | level=info ts=2022-07-26T18:34:46.703031991Z caller=compactor.go:418 msg="compactor startup delay completed"
loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | level=info ts=2022-07-26T18:34:46.703134833Z caller=compactor.go:469 msg="compactor started"
loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | ts=2022-07-26T18:34:46.703191697Z caller=spanlogger.go:80 level=info msg="building index list cache"
loki4_loki-read.1.o52033t8oat6@srv-swarm-worker3.infra.ginhoux.net    | ts=2022-07-26T18:34:46.716529969Z caller=spanlogger.go:80 level=info msg="index list cache built" duration=13.27361ms

Logs from write service : 

loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:43.713991016Z caller=lifecycler.go:547 msg="not loading tokens from file, tokens file path is empty"
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:43.714195986Z caller=lifecycler.go:576 msg="instance not found in ring, adding with no tokens" ring=distributor
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:43.714360692Z caller=lifecycler.go:416 msg="auto-joining cluster after timeout" ring=distributor
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:43.724763841Z caller=ingester.go:417 msg="recovered WAL checkpoint recovery finished" elapsed=11.508013ms errors=false
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:43.724808179Z caller=ingester.go:423 msg="recovering from WAL"
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | ts=2022-07-26T18:24:43.729712178Z caller=memberlist_logger.go:74 level=warn msg="Failed to resolve loki-write: lookup loki-write on 127.0.0.11:53: no such host"
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:43.729766645Z caller=memberlist_client.go:534 msg="joined memberlist cluster" reached_nodes=1
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:43.73439661Z caller=ingester.go:439 msg="WAL segment recovery finished" elapsed=21.140855ms errors=false
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:43.734929872Z caller=ingester.go:387 msg="closing recoverer"
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:43.735000819Z caller=ingester.go:395 msg="WAL recovery finished" time=21.745559ms
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:43.735219811Z caller=lifecycler.go:547 msg="not loading tokens from file, tokens file path is empty"
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:43.735307214Z caller=lifecycler.go:576 msg="instance not found in ring, adding with no tokens" ring=ingester
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:43.735481093Z caller=wal.go:156 msg=started component=wal
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:43.740704611Z caller=loki.go:374 msg="Loki started"
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | ts=2022-07-26T18:24:53.710136343Z caller=memberlist_logger.go:74 level=info msg="Suspect a24c63868dd7-52e08473 has failed, no acks received"
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:53.735551338Z caller=lifecycler.go:416 msg="auto-joining cluster after timeout" ring=ingester
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:53.735791542Z caller=lifecycler.go:425 msg="observing tokens before going ACTIVE" ring=ingester
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:58.736802354Z caller=lifecycler.go:444 msg="token verification successful" ring=ingester
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:24:58.736842733Z caller=lifecycler.go:791 msg="changing instance state from" old_state=JOINING new_state=ACTIVE ring=ingester
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | ts=2022-07-26T18:25:08.710190551Z caller=memberlist_logger.go:74 level=info msg="Suspect a24c63868dd7-52e08473 has failed, no acks received"
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | ts=2022-07-26T18:25:13.710622976Z caller=memberlist_logger.go:74 level=info msg="Marking a24c63868dd7-52e08473 as failed, suspect timeout reached (0 peer confirmations)"
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | ts=2022-07-26T18:25:28.70936503Z caller=memberlist_logger.go:74 level=info msg="Suspect a24c63868dd7-52e08473 has failed, no acks received"
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:25:43.671292354Z caller=table_manager.go:167 msg="handing over indexes to shipper"
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:25:43.671351734Z caller=table_manager.go:134 msg="uploading tables"
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:26:43.670797877Z caller=table_manager.go:134 msg="uploading tables"
loki4_loki-write.1.uhkm1alaowkt@srv-swarm-worker2.infra.ginhoux.net    | level=info ts=2022-07-26T18:26:43.672869348Z caller=table_manager.go:167 msg="handing over indexes to shipper"

NOTE : i also work on mimir and have similar issue, with the ring !

NOTE2 : i've tried to chnage both hostname or add alias on both service / config file.... no more positive result.

sungaohui commented 2 years ago

you using a loki-read and a loki-write，can‘t see the error. I use three read and three write, the error "caller=scheduler_processor.go:86 msg="error contacting scheduler" err="rpc error: code = Canceled desc = context canceled"". should add config for component scheduler health check time

dginhoux commented 2 years ago

Hi,

Do you use swarm like me? What can be different between our both stack and config?

saibug commented 2 years ago

Hi @dginhoux From your config you're using Minio s3 storage, and it based on filesystem with only one node! So you cannot use loki distributed method without shared storage like AWS, GCP S3 object store . You can test with this simple example of scalable method of loki https://gist.github.com/wardbekker/6abde118f530a725e60acb5adb04508a

Regards

dginhoux commented 10 months ago

Hi grafana devs teams !

Can you test loki in a swarm cluster with distributed and scaled services and provide a working example ?

cec commented 10 months ago

Hi,

I spent a fair amount of time configuring the whole lgtm stack in swarm with scalable services.

Despite reaching an almost productive setup, the plethora of issues I encountered due to the limitations of swarm lead me to stop and move to a kubernetes setup.

If your organization is not yet on kubernetes, I would suggest to get a partnership with a consulting agency to setup a manages kubernetes for you.

I know it has drawbacks, but I find this to be the best solution.

If you are interested I can share my configuration for Loki.

Unless Grafana people provide a working maintained configuration of course.

Il dom 31 dic 2023, 12:16 Dany GINHOUX @.***> ha scritto:

Hi grafana devs teams !

Can you test loki in a swarm cluster with distributed and scaled services and provide a working example ?

— Reply to this email directly, view it on GitHub https://github.com/grafana/loki/issues/6648#issuecomment-1872923571, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC6I2Y7EKEYVK3ZQEEGAZLYMFCPBAVCNFSM53CLD2P2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBXGI4TEMZVG4YQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

pcace commented 9 months ago 
version: '3.8'

services:
  loki:
    image: grafana/loki:2.6.1
    command: >-
      -config.file=/etc/loki/local-config.yaml
      -boltdb.shipper.compactor.ring.instance-addr=127.0.0.1
      -distributor.ring.instance-addr=127.0.0.1
      -frontend.instance-addr=127.0.0.1
      -index-gateway.ring.instance-addr=127.0.0.1
      -ingester.lifecycler.addr=127.0.0.1
      -query-scheduler.ring.instance-addr=127.0.0.1
      -ruler.ring.instance-addr=127.0.0.1
    ports:
      - 3100:3100
    volumes:
      - data:/loki

volumes:
  data: {}

Hi there, can i integrate this somehow in the local-config.yaml?

Thanks a lot