Docker Swarm Cluster - Unable to initialize catalog manager

[fabiocmazzo]

Hello, I'm trying to start a 3 node YugabyteDB in DockerSwarm with 2 Nodes with this "Stack Docker Compose":

version: '3.6'
services:
   yugabyte-master1:
    image: yugabytedb/yugabyte:1.0.5.2-b4
    command: 
    - "/home/yugabyte/bin/yb-master"
    - "--fs_data_dirs=/mnt/data0"
    - "--master_addresses=yugabyte-master1:7100,yugabyte-master2:7100,yugabyte-master3:7100"
    - "--replication_factor=3"
    ports:
    - 7000:7000
    deploy:
      replicas: 1
    networks:
    - web
    volumes:
    - type: volume
      source: yugabytevol1
      target: /mnt/data0

   yugabyte-master2:
    image: yugabytedb/yugabyte:1.0.5.2-b4
    command: 
    - "/home/yugabyte/bin/yb-master"
    - "--fs_data_dirs=/mnt/data0"
    - "--master_addresses=yugabyte-master1:7100,yugabyte-master2:7100,yugabyte-master3:7100"
    - "--replication_factor=3"
    networks:
    - web
    deploy:
      replicas: 1
    volumes:
    - type: volume
      source: yugabytevol2
      target: /mnt/data0

   yugabyte-master3:
    image: yugabytedb/yugabyte:1.0.5.2-b4
    command: 
    - "/home/yugabyte/bin/yb-master"
    - "--fs_data_dirs=/mnt/data0"
    - "--master_addresses=yugabyte-master1:7100,yugabyte-master2:7100,yugabyte-master3:7100"
    - "--replication_factor=3"
    networks:
    - web
    deploy:
      replicas: 1
    volumes:
    - type: volume
      source: yugabytevol3
      target: /mnt/data0

volumes:
  yugabytevol1:
  yugabytevol2:
  yugabytevol3:

networks:
  web:
    external: true

And i'm getting this error in all services: Not found (yb/util/env_posix.cc:1011): Unable to initialize catalog manager: Failed to initialize sys tables async: Could not load tablet metadata from /mnt/data0/yb-data/master/tablet-meta/00000000000000000000000000000000: /mnt/data0/yb-data/master/tablet-meta/00000000000000000000000000000000: No such file or directory (error 2)

Thank You!

[schoudhury]

Fabio,

1. Our docker-compose instructions are here

2. Our docker swarm mode instructions are here

Since we have not tested yugabyte db with docker stack yet, it may take us a few days to figure out what is wrong with the above yml. Meanwhile, hoping you can use either 1 or 2 above to get unblocked.

[fabiocmazzo]

@schoudhury Thanks for the quick response,

 I'm trying start a cluster in my local machine - to help me understanding last error - using yb-docker-ctl following Quick Start tutorial, and i'm facing this error:

Illegal state (yb/master/catalog_manager.cc:1182): Unable to initialize catalog manager: Failed to initialize sys tables async: Local host/port 172.18.0.2:7100 not present in master_addresses yb-master-n1:7100,yb-master-n2:7100,yb-master-n3:7100.

I think maybe this error might be related to my version of Docker. I'm using Docker version 18.03.1-ce, build 9ee9f40

Could you make Dockerfile the latest tag available? So I can try to evaluate better.

Thank you so much,

PS: I am very happy with the Yugabyte initiative and I want to help in whatever way possible.

[schoudhury]

Hi @fabiocmazzo, thanks for your support for the yugabyte db project! we are investigating the issue you reported and will release a fix soon. meanwhile, you can edit the yb-docker-ctl script to use the docker image tag 1.0.4.0-b24 instead of latest -- the issue you reported does not exist in that image.

[bbaddepudi]

Thanks for the feedback @fabiomazzo, appreciate it!

For this issue, we are looking into hostname resolution causing ip address to be used to check against master addresses hostname list. We are working on a generic changes also for long term, will get this prioritized soon.

[fabiocmazzo]

Hi @schoudhury with b24 tag this Swarm Stack Compose works like a charm:

version: '3.6'
services:
   yugabyte-master1:
    image: yugabytedb/yugabyte:1.0.4.0-b24
    command: 
    - "/home/yugabyte/bin/yb-master"
    - "--fs_data_dirs=/mnt/data0" 
    - "--master_addresses=yugabyte-master1:7100,yugabyte-master2:7150,yugabyte-master3:7200"
    - "--replication_factor=3"
    - "--rpc_bind_addresses=0.0.0.0:7100"
    ports:
    - 7000:7000
    networks:
    - web
    volumes:
    - yugabytevol1:/mnt/data0/

   yugabyte-master2:
    image: yugabytedb/yugabyte:1.0.4.0-b24
    command: 
    - "/home/yugabyte/bin/yb-master"
    - "--fs_data_dirs=/mnt/data0"
    - "--master_addresses=yugabyte-master1:7100,yugabyte-master2:7150,yugabyte-master3:7200"
    - "--replication_factor=3"
    - "--rpc_bind_addresses=0.0.0.0:7150"

    networks:
    - web
    volumes:
    - yugabytevol2:/mnt/data0/

   yugabyte-master3:
    image: yugabytedb/yugabyte:1.0.4.0-b24
    command: 
    - "/home/yugabyte/bin/yb-master"
    - "--fs_data_dirs=/mnt/data0"
    - "--master_addresses=yugabyte-master1:7100,yugabyte-master2:7150,yugabyte-master3:7200"
    - "--replication_factor=3"
    - "--rpc_bind_addresses=0.0.0.0:7200"

    networks:
    - web
    volumes:
    - yugabytevol3:/mnt/data0/

   yugabyte-tserver:
    image: yugabytedb/yugabyte:1.0.4.0-b24
    command:
    - "/home/yugabyte/bin/yb-tserver"
    - "--fs_data_dirs=/mnt/data0"
    - "--tserver_master_addrs=yugabyte-master1:7100,yugabyte-master2:7150,yugabyte-master3:7200"
    - "--tserver_master_replication_factor=3"
    deploy:
      mode: global
    networks:
    - web
    ports:
    - 9990:9000
    volumes:
    - yugabytetserver:/mnt/data0

volumes:
  yugabytevol1:
  yugabytevol2:
  yugabytevol3:
  yugabytetserver:

networks:
  web:
    external: true