search

rqlite / rqlite-docker

rqlite docker images
MIT License
25 stars 11 forks source link

Rqlited does not start in docker swarm mode #9

Open sginer opened 5 years ago

sginer commented 5 years ago

Rqlited service does not start in docker service.

When I use the docker definition of the 4.5.0 version with my own docker-entrypoint.sh with this content

#!/bin/bash

if [ "x" == "x${JOIN_ADRESS}" ]; then
        exec rqlited -http-addr ${HOSTNAME}:4001  /rqlite/file/data
else
        exec rqlited -http-addr ${HOSTNAME}:4001  -join http://${JOIN_ADRESS}:4001  /rqlite/file/data
fi

and when I launch it with this compose file whith the cmd docker stack deploy -c docker-compose.yml rqlite

version: "3.3"

networks:
  rqlite:

services:
 rqlite1:
    image: sginer/rqlite:4.5.0
    environment:
      NODE_ID: "1"
      HOSTNAME: "rqlite1"
    networks:
      - rqlite

  rqlite2:
    image: sginer/rqlite:4.5.0
    environment:
      NODE_ID: "2"
      HOSTNAME: "rqlite2"
      JOIN_ADRESS: "rqlite1"
    networks:
      - rqlite

 rqlite3:
    image: sginer/rqlite:4.5.0
    environment:
      NODE_ID: "3"
      HOSTNAME: "rqlite3"
      JOIN_ADRESS: "rqlite1"
    networks:
      - rqlite

rqlite is unable to elect leader and to start !

otoolep commented 5 years ago

Can you supply any logs, from the nodes? Each node should be printing a bunch of output on startup.

sginer commented 5 years ago

Docker version

Client: Version: 18.09.4 API version: 1.39 Go version: go1.10.8 Git commit: d14af54266 Built: Wed Mar 27 18:35:44 2019 OS/Arch: linux/amd64 Experimental: false

Server: Docker Engine - Community Engine: Version: 18.09.4 API version: 1.39 (minimum version 1.12) Go version: go1.10.8 Git commit: d14af54 Built: Wed Mar 27 18:01:48 2019 OS/Arch: linux/amd64 Experimental: false

Logs of master

rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [rqlited] 2019/10/09 14:45:07 rqlited starting, version v4.5.0, commit 8336150318dfb2b1f196f6a4919041b65071f3fd, branch 4.3.0-patch
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [rqlited] 2019/10/09 14:45:07 go1.10, target architecture is amd64, operating system target is linux
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |             _ _ _
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |            | (_) |
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |   _ __ __ _| |_| |_ ___
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |  | '__/ _  | | | __/ _ \   The lightweight, distributed
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |  | | | (_| | | | ||  __/   relational database.
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |  |_|  \__, |_|_|\__\___|
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |          | |
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |          |_|
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [store] 2019/10/09 14:45:07 ensuring /rqlite/file/data exists
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [mux] 2019/10/09 14:45:07 mux serving on 127.0.0.1:4002, advertising 127.0.0.1:4002
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [store] 2019/10/09 14:45:07 SQLite in-memory database opened
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [store] 2019/10/09 14:45:07 enabling single-node mode
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [store] 2019/10/09 14:45:07 waiting for up to 2m0s for application of initial logs
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:07 [INFO] raft: Node at 127.0.0.1:4002 [Follower] entering Follower state (Leader: "")
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [cluster] 2019/10/09 14:45:07 service listening on 127.0.0.1:4002
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [rqlited] 2019/10/09 14:45:07 no join addresses set
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [rqlited] 2019/10/09 14:45:08 failed to set peer for 127.0.0.1:4002 to rqlite1:4001: no leader available (retrying)
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:08 [WARN] raft: Heartbeat timeout from "" reached, starting election
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:08 [INFO] raft: Node at 127.0.0.1:4002 [Candidate] entering Candidate state
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:08 [DEBUG] raft: Votes needed: 1
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:08 [DEBUG] raft: Vote granted from 127.0.0.1:4002. Tally: 1
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:08 [INFO] raft: Election won. Tally: 1
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:08 [INFO] raft: Node at 127.0.0.1:4002 [Leader] entering Leader state
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:08 [DEBUG] raft: Node 127.0.0.1:4002 updated peer set (2): [127.0.0.1:4002]
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [rqlited] 2019/10/09 14:45:09 set peer for 127.0.0.1:4002 to rqlite1:4001
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [rqlited] 2019/10/09 14:45:09 failed to start HTTP server: listen tcp 10.0.7.2:4001: bind: cannot assign requested address

Logs of followers

rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | 
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |             _ _ _
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |            | (_) |
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |   _ __ __ _| |_| |_ ___
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |  | '__/ _  | | | __/ _ \   The lightweight, distributed
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |  | | | (_| | | | ||  __/   relational database.
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |  |_|  \__, |_|_|\__\___|
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |          | |
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |          |_|
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | 
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [rqlited] 2019/10/10 06:15:26 rqlited starting, version v4.5.0, commit 8336150318dfb2b1f196f6a4919041b65071f3fd, branch 4.3.0-patch
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [rqlited] 2019/10/10 06:15:26 go1.10, target architecture is amd64, operating system target is linux
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [store] 2019/10/10 06:15:26 ensuring /rqlite/file/data exists
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [mux] 2019/10/10 06:15:26 mux serving on 127.0.0.1:4002, advertising 127.0.0.1:4002
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [store] 2019/10/10 06:15:26 SQLite in-memory database opened
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [store] 2019/10/10 06:15:26 waiting for up to 2m0s for application of initial logs
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | 2019/10/10 06:15:26 [INFO] raft: Node at 127.0.0.1:4002 [Follower] entering Follower state (Leader: "")
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [cluster] 2019/10/10 06:15:26 service listening on 127.0.0.1:4002
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [rqlited] 2019/10/10 06:15:26 join addresses are: [http://rqlite1:4001]
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [cluster-join] 2019/10/10 06:15:26 failed to join cluster at [http://rqlite1:4001], sleeping 5s before retry
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | 2019/10/10 06:15:27 [WARN] raft: EnableSingleNode disabled, and no known peers. Aborting election.
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [cluster-join] 2019/10/10 06:15:31 failed to join cluster at [http://rqlite1:4001], sleeping 5s before retry
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [cluster-join] 2019/10/10 06:15:36 failed to join cluster at [http://rqlite1:4001], sleeping 5s before retry
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [cluster-join] 2019/10/10 06:15:41 failed to join cluster at [http://rqlite1:4001], after 3 attempts
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [rqlited] 2019/10/10 06:15:41 failed to join cluster at [http://rqlite1:4001]: Post http://rqlite1:4001/join: dial tcp 10.0.8.5:4001: connect: connection refused
rqlite_rqlite2.1.6c4x4bmtu00p@ubuntu-VirtualBox    | [rqlited] 2019/10/10 06:15:49 rqlited starting, version v4.5.0, commit 8336150318dfb2b1f196f6a4919041b65071f3fd, branch 4.3.0-patch
rqlite_rqlite2.1.6c4x4bmtu00p@ubuntu-VirtualBox    | [rqlited] 2019/10/10 06:15:49 go1.10, target architecture is amd64, operating system target is linux
rqlite_rqlite2.1.6c4x4bmtu00p@ubuntu-VirtualBox    | [store] 2019/10/10 06:15:49 ensuring /rqlite/file/data exists
rqlite_rqlite2.1.6c4x4bmtu00p@ubuntu-VirtualBox    | [store] 2019/10/10 06:15:49 SQLite in-memory database opened
rqlite_rqlite2.1.6c4x4bmtu00p@ubuntu-VirtualBox    | [mux] 2019/10/10 06:15:49 mux serving on 127.0.0.1:4002, advertising 127.0.0.1:4002
otoolep commented 5 years ago

Something is wrong with your network setup:

rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox | [rqlited] 2019/10/09 14:45:09 failed to start HTTP server: listen tcp 10.0.7.2:4001: bind: cannot assign requested address

sginer commented 5 years ago

I'm agree with you. Is raft use a spécific protocol or need a specific network configuration (like multicast or other). Have you ever test rqlite in a swarm environnement ? When I use it on a single node with docker compose all work fine.

otoolep commented 5 years ago

I have not tested it in a Swarm environment. An rqlite node opens up two TCP ports -- one for internode Raft communication, and the other for the HTTP server.

I suggest that either that address/port is in use, or is not even visible to the rqlite node.

otoolep commented 5 years ago

This is also a sign of problems: rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox | [cluster-join] 2019/10/10 06:15:26 failed to join cluster at [http://rqlite1:4001], sleeping 5s before retry

otoolep commented 5 years ago

Perhaps you should mail the rqlite Google Group, someone there may be able to help.

mikeTWC1984 commented 9 months ago

After struggling for a while I got it to work using official image:

version: '3.8'

services:
  rqlite1:
    hostname: 'RQX{{.Task.Slot}}'
    image: rqlite/rqlite
    command: [ "-join=RQX1:4002,RQX2:4002,RQX3:4002", "-bootstrap-expect=3" ]
    deploy:
      replicas: 3
    ports:
      - "4001:4001"
    #   - "4002:4002"
    networks:
      - main
networks:
  main:
    external: true