FranckPachot
commented
1 year ago Thanks a lot, @pablopla for filling the issue and identifying the root cause. Yes, YugabyteDB needs a static network identity for its nodes, with a static IP. That's why in Kubernetes it is deployed with StatefulSets. For labs, can do something similar with Docker network.
Reproduce without restarting the host
By starting another container when the YugabyteDB one is stopped, it takes the IP address and then the next start will get a new one assigned:
docker run -it -d --restart=always -p 5433:5433 --hostname yugabyte --name yugabyte yugabytedb/yugabyte:2.18.1.0-b84 bin/yugabyted start --daemon=false
docker exec yugabyte bash -c 'until postgres/bin/pg_isready -h yugabyte ; do sleep 1 ; done | uniq'
docker exec -i yugabyte bash -c 'ysqlsh -h $(hostname)' <<<'create table franck as select 42;'
docker exec yugabyte hostname -i
# stopping the container releases the IP
docker stop yugabyte
# start another container that will take the IP
docker run -d --name zzz alpine sleep infinity
docker exec zzz hostname -i
# If the IP is the same, starting yugabyte will get a new one but tries to bind to the old one
docker start yugabyte
docker logs yugabyte
# cleanup
docker rm -f yugabyte zzzThe error is:
Failed to bind to address: 172.17.0.2:7100:
For more information, check the logs in /root/var/logsbecause this address is already used by another container.
Solution: Static IP
What is different here is that I add --network yb42 --ip 172.42.10.1 on docker run to attach with a static IP address to a network that I've created with docker network create --subnet=172.42.0.0/16 yb42.
docker network create --subnet=172.42.0.0/16 yb42
docker run -it -d --restart=always -p 5433:5433 --hostname yugabyte --network yb42 --name yugabyte yugabytedb/yugabyte:2.18.1.0-b84 bin/yugabyted start --daemon=false
docker exec yugabyte bash -c 'until postgres/bin/pg_isready -h yugabyte ; do sleep 1 ; done | uniq'
docker exec -i yugabyte bash -c 'ysqlsh -h $(hostname)' <<<'create table franck as select 42;'
docker exec yugabyte hostname -i
# stopping the container releases the IP
docker stop yugabyte
# start another container that will take the IP
docker run -d --name zzz alpine sleep infinity
docker exec zzz hostname -i
# restart will use the same parameters with the static IP address
docker start yugabyte
docker logs yugabyte
# check that my table is still there
docker exec -i yugabyte bash -c 'ysqlsh -h yugabyte.yb42' <<<'\d'
# cleanup
docker rm -f yugabyte zzzOf course, if another container is started on the same network without setting another IP address, we will encounter the same problem. But now we have full control of the IP assigned.
pablopla
Jira Link: DB-7509
Description
Yugabyte container fail to bind socket after restarting the host. Docker assign IP addresses to containers based on the start order. If you have several containers running on your machine and restart the host, the Yugabyte container might get a different address.
To reproduce:
sudo docker run -it -d --restart=always -p 5433:5433 --hostname yugabyte --name yugabyte yugabytedb/yugabyte:2.18.1.0-b84 bin/yugabyted start --daemon=falsesudo docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' yugabytesudo docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' yugabytecat /root/var/logs/tserver.err20 tablet_server_main_impl.cc:208] Network error (yb/util/net/socket.cc:325): Error binding socket to 172.19.0.7:9100: Cannot assign requested address (system error 99)Warning: Please confirm that this issue does not contain any sensitive information