search

xinity / pxc_swarm

a docker swarm Percona PXC setup
Apache License 2.0
36 stars 12 forks source link

etcd bootstrap - failed resolving host galera_etcd:2380 #5

Closed worp1900 closed 6 years ago

worp1900 commented 6 years ago

Hey man,

an amazing tutorial. I was able to follow through it just perfectly. I got it all running smoothly on a local 3-node cluster.

However when I went on to our testing environment to deploy the stack there, etcd kept reporting:

percona_etcd.1.pu3dhbwh5tdx@test-011    | 2018-04-24 18:26:05.857945 E | pkg/netutil: could not resolve host galera_etcd:2380
percona_etcd.1.pu3dhbwh5tdx@test-011    | 2018-04-24 18:26:05.858792 C | etcdmain: failed to resolve http://galera_etcd:2380 to match --initial-cluster=etcd0=http://galera_etcd:2380 (failed to resolve "http://galera_etcd:2380" (lookup galera_etcd on 127.0.0.11:53: no such host))
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499244 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://galera_etcd:2379,http://galera_etcd:4001
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499300 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/opt/etcd/data
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499321 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://galera_etcd:2380
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499330 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd0=http://galera_etcd:2380
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499334 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499339 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499347 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,http://0.0.0.0:4001
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499354 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499367 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd0
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499411 I | etcdmain: etcd Version: 3.3.3
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499417 I | etcdmain: Git SHA: e348b1aed
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499420 I | etcdmain: Go Version: go1.9.5
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499423 I | etcdmain: Go OS/Arch: linux/amd64
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499426 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499496 I | embed: listening for peers on http://0.0.0.0:2380
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499522 I | embed: listening for client requests on 0.0.0.0:2379
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.499544 I | embed: listening for client requests on 0.0.0.0:4001
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:11.502371 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:12.503543 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:13.504880 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:14.506266 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:15.507648 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:16.509034 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:17.510374 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:18.511540 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:19.512725 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:20.513914 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:21.515033 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:22.516254 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:23.517597 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:24.518835 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:25.520160 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:26.521383 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:27.522600 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:28.523986 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:29.525255 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:30.526946 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:31.528323 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:32.529844 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:33.531065 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:34.532294 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:35.533496 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:36.534812 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:37.536152 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:38.537574 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:39.538832 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:40.540126 W | pkg/netutil: failed resolving host galera_etcd:2380 (lookup galera_etcd on 127.0.0.11:53: no such host); retrying in 1s
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:41.501414 E | pkg/netutil: could not resolve host galera_etcd:2380
percona_etcd.1.srj6veewwulx@test-011    | 2018-04-24 18:26:41.502104 C | etcdmain: failed to resolve http://galera_etcd:2380 to match --initial-cluster=etcd0=http://galera_etcd:2380 (failed to resolve "http://galera_etcd:2380" (lookup galera_etcd on 127.0.0.11:53: no such host))

I assume there is a problem resolving DNS names? I have no clue where I might find help with that as the search for "etcd failed resolving hostname" or "etcdmain failed to resolve" does not yield anything viable yet. I am also thinking it's not an etcd issue but a Docker issue. Maybe it's not connected to DNS but maybe I am missing a port to open?

If you can provide any hint I am happy for anything!

worp1900 commented 6 years ago

I was able to fix etcd not coming up by adjusting the docker-compose file like so:

etcd:
    image: quay.io/coreos/etcd

--> hostname: galera_etcd command: etcd [...]

Now the xtradb instances have problems connecting to galera_etcd:

percona_percona-xtradb-cluster.1.6n00gfc8zbdr@test-011    | Registering in the discovery service
percona_percona-xtradb-cluster.1.6n00gfc8zbdr@test-011    |
percona_percona-xtradb-cluster.1.6n00gfc8zbdr@test-011    |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
percona_percona-xtradb-cluster.2.o04vqqx8hpds@test-010    |
percona_percona-xtradb-cluster.1.6n00gfc8zbdr@test-011    |                                  Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0curl: (6) Could not resolve host: galera_etcd
percona_percona-xtradb-cluster.1.6n00gfc8zbdr@test-011    |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
percona_percona-xtradb-cluster.2.o04vqqx8hpds@test-010    | MySQL init process done. Ready for start up.
percona_percona-xtradb-cluster.1.6n00gfc8zbdr@test-011    |                                  Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0curl: (6) Could not resolve host: galera_etcd
percona_percona-xtradb-cluster.2.o04vqqx8hpds@test-010    |
percona_percona-xtradb-cluster.2.o04vqqx8hpds@test-010    |
percona_percona-xtradb-cluster.1.6n00gfc8zbdr@test-011    |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
percona_percona-xtradb-cluster.1.6n00gfc8zbdr@test-011    |                                  Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0curl: (6) Could not resolve host: galera_etcd
percona_percona-xtradb-cluster.1.6n00gfc8zbdr@test-011    |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

percona_percona-xtradb-cluster.2.o04vqqx8hpds@test-010    | Registering in the discovery service
percona_percona-xtradb-cluster.2.o04vqqx8hpds@test-010    |
percona_percona-xtradb-cluster.1.6n00gfc8zbdr@test-011    |                                  Dload  Upload   Total   Spent    Left  Speed
percona_percona-xtradb-cluster.2.o04vqqx8hpds@test-010    |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0curl: (6) Could not resolve host: galera_etcd
percona_percona-xtradb-cluster.1.6n00gfc8zbdr@test-011    |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
percona_percona-xtradb-cluster.1.6n00gfc8zbdr@test-011    |                                  Dload  Upload   Total   Spent    Left  Speed
percona_percona-xtradb-cluster.2.o04vqqx8hpds@test-010    |                                  Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0curl: (6) Could not resolve host: galera_etcd
percona_percona-xtradb-cluster.1.6n00gfc8zbdr@test-011    |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0curl: (6) Could not resolve host: galera_etcd
percona_percona-xtradb-cluster.1.6n00gfc8zbdr@test-011    |                                  Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0curl: (6) Could not resolve host: galera_etcd
percona_percona-xtradb-cluster.1.6n00gfc8zbdr@test-011    | Joining cluster
[...]
percona_percona-xtradb-cluster.2.o04vqqx8hpds@test-010    |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
percona_percona-xtradb-cluster.2.o04vqqx8hpds@test-010    |                                  Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0curl: (6) Could not resolve host: galera_etcd
percona_percona-xtradb-cluster.2.o04vqqx8hpds@test-010    |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
percona_percona-xtradb-cluster.2.o04vqqx8hpds@test-010    |                                  Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0curl: (6) Could not resolve host: galera_etcd
percona_percona-xtradb-cluster.2.o04vqqx8hpds@test-010    | Joining cluster

It is probably fairly easy...I just can't figure out what's going wrong. Obviously the services can't resolve the service via the docker overlay network.

Any hints to debug this issue or ideas how to proceed are greatly appreciated!

worp1900 commented 6 years ago

Turns out I had a rename done wrong, renaming the original "galera_etcd" from your tutorial to "percona_etcd", since I had changed the cluster-name to "percona". This skipped my mind.