Closed criscola closed 1 year ago
@criscola: This issue is currently awaiting triage.
If CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted
label and provide further guidance.
The triage/accepted
label can be added by org members by writing /triage accepted
in a comment.
/triage needs-information
@criscola Are you using docker installed with snap? we got report in the past about strange behaviour when using docker installed from snap.
Otherwise, we need some more info to investigate whats going on, because it doesn't happen nor on CI nor on our dev envs.
Can we get the docker inspect of the container not reporting the ip?
What we do when reading ips is ~equal to
docker inspect your-contatiner | jq '.[0].NetworkSettings.Networks["kind"] | { ipv4: .IPAddress, ipv6: .GlobalIPv6Address }'
Note: kind is the only network in our containers
Hi Fabrizio, docker was installed using pacman
.
To precise, the lb container doesn't get brought up for an already created cluster after booting my workstation, so I can't get any IP. Afterwards, if I recreate the cluster the container is also not created until the 2nd cluster resource application. After that it finally gets created and the CAPD cluster works:
{
"ipv4": "172.18.0.2",
"ipv6": "fc00:f853:ccd:e793::2"
}
In the meantime I tried getting some logs immediately after booting up my machine with journalctl -fu docker.service
and I get the following:
Dec 20 13:10:38 lxray dockerd[1165]: time="2022-12-20T13:10:38.230504583+01:00" level=error msg="Error setting up exec command in container mycluster-dev-lb: Container 9427845ce3fded0d4e8115a678b2eeebb3feee3c86785bdeb107f3d2e6518788 is not running"
so it seems like the container is for some reason, unable to run. This is different than the containers for the CAPD nodes which are brought up without issues.
@criscola can you try running a haproxy container outside of CAPD? This is how I managed to debug similar issues in the past -
The command I was using was:
docker run -v $(pwd)/images/haproxy:/usr/local/etc/conf:ro haproxy:2.6 -f /usr/local/etc/conf/haproxy.cfg -d -V
Where $(pwd)/images/haproxy
points to a directory containing a valid haproxy cfg file called haproxy.cfg. In my case there was a bug with Fedora wherre haproxy would start, use up all available resources, and then crash.
Do you have a working haproxy.cfg file around?
Apologies - you can use the cfg in the kind repo at https://github.com/kubernetes-sigs/kind/tree/main/images/haproxy.
Thanks, I was struggling to find one. Here's what I get (I run it after rebooting so when it should have been brought up):
[NOTICE] (1) : haproxy version is 2.6.7-c55bfdb
[NOTICE] (1) : path to executable is /usr/local/sbin/haproxy
[WARNING] (1) : config : missing timeouts for frontend 'controlPlane'.
| While not properly invalid, you will certainly encounter various problems
| with such a configuration. To fix this, please ensure that all following
| timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Note: setting global.maxconn to 524241.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result FAILED
Total: 3 (2 usable), will use epoll.
Available filters :
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
Using epoll() as the polling mechanism.
[NOTICE] (1) : New worker (8) forked
[NOTICE] (1) : Loading success.
Using epoll() as the polling mechanism.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
That looks pretty okay / as expected too me - it's similar to what I see on my machine. Can you retrieve the logs or the failure reason for the haproxy container started by CAPD? Docker inspect might give you some information for how and why the lb container failed.
Here it is:
[WARNING] 353/131805 (1) : config : missing timeouts for frontend 'controlPlane'.
| While not properly invalid, you will certainly encounter various problems
| with such a configuration. To fix this, please ensure that all following
| timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[NOTICE] 353/131805 (1) : New worker #1 (8) forked
[WARNING] 353/131809 (1) : Reexecuting Master process
[NOTICE] 353/131809 (1) : haproxy version is 2.2.9-2~bpo10+1
[NOTICE] 353/131809 (1) : path to executable is /usr/sbin/haproxy
[ALERT] 353/131809 (1) : sendmsg()/writev() failed in logger #1: No such file or directory (errno=2)
[WARNING] 353/131809 (8) : Stopping frontend controlPlane in 0 ms.
[WARNING] 353/131809 (8) : Stopping backend kube-apiservers in 0 ms.
[WARNING] 353/131809 (8) : Stopping frontend GLOBAL in 0 ms.
[WARNING] 353/131809 (8) : Proxy controlPlane stopped (cumulated conns: FE: 6, BE: 0).
[WARNING] 353/131809 (8) : Proxy kube-apiservers stopped (cumulated conns: FE: 0, BE: 6).
[WARNING] 353/131809 (8) : Proxy GLOBAL stopped (cumulated conns: FE: 0, BE: 0).
[NOTICE] 353/131809 (1) : New worker #1 (45) forked
[WARNING] 353/131809 (1) : Former worker #1 (8) exited with code 0 (Exit)
[WARNING] 353/131809 (45) : Server kube-apiservers/ecoqube-dev-hmhsb-xnw4b is DOWN, reason: Layer4 connection problem, info: "SSL handshake failure", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 353/131809 (45) : backend 'kube-apiservers' has no server available!
[WARNING] 353/131828 (45) : Server kube-apiservers/ecoqube-dev-hmhsb-xnw4b is UP, reason: Layer7 check passed, code: 200, check duration: 3ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING] 353/132448 (1) : Exiting Master process...
[WARNING] 353/132448 (45) : Stopping frontend control-plane in 0 ms.
[WARNING] 353/132448 (45) : Stopping backend kube-apiservers in 0 ms.
[WARNING] 353/132448 (45) : Stopping frontend GLOBAL in 0 ms.
[WARNING] 353/132448 (45) : Proxy control-plane stopped (cumulated conns: FE: 207, BE: 0).
[WARNING] 353/132448 (45) : Proxy kube-apiservers stopped (cumulated conns: FE: 0, BE: 207).
[WARNING] 353/132448 (45) : Proxy GLOBAL stopped (cumulated conns: FE: 0, BE: 0).
[WARNING] 353/132449 (1) : Current worker #1 (45) exited with code 0 (Exit)
[WARNING] 353/132449 (1) : All workers exited. Exiting... (0)
Couldn't it retry again instead of just exiting? Why is there a different behavior for the haproxy at boot time vs after? That's the log for when it stays up:
[WARNING] 353/134529 (1) : config : missing timeouts for frontend 'controlPlane'.
| While not properly invalid, you will certainly encounter various problems
| with such a configuration. To fix this, please ensure that all following
| timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[NOTICE] 353/134529 (1) : New worker #1 (8) forked
[WARNING] 353/134531 (1) : Reexecuting Master process
[NOTICE] 353/134531 (1) : haproxy version is 2.2.9-2~bpo10+1
[NOTICE] 353/134531 (1) : path to executable is /usr/sbin/haproxy
[ALERT] 353/134531 (1) : sendmsg()/writev() failed in logger #1: No such file or directory (errno=2)
[WARNING] 353/134531 (8) : Stopping frontend controlPlane in 0 ms.
[WARNING] 353/134531 (8) : Stopping backend kube-apiservers in 0 ms.
[WARNING] 353/134531 (8) : Stopping frontend GLOBAL in 0 ms.
[WARNING] 353/134531 (8) : Proxy controlPlane stopped (cumulated conns: FE: 6, BE: 0).
[WARNING] 353/134531 (8) : Proxy kube-apiservers stopped (cumulated conns: FE: 0, BE: 6).
[WARNING] 353/134531 (8) : Proxy GLOBAL stopped (cumulated conns: FE: 0, BE: 0).
[NOTICE] 353/134531 (1) : New worker #1 (45) forked
[WARNING] 353/134531 (1) : Former worker #1 (8) exited with code 0 (Exit)
[WARNING] 353/134532 (45) : Server kube-apiservers/ecoqube-dev-bbqhg-42n7f is DOWN, reason: Layer4 connection problem, info: "SSL handshake failure", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 353/134532 (45) : backend 'kube-apiservers' has no server available!
[WARNING] 353/134551 (45) : Server kube-apiservers/ecoqube-dev-bbqhg-42n7f is UP, reason: Layer7 check passed, code: 200, check duration: 2ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
this time the haproxy doesn't exit and the cluster can be initialized, seems weird to me that it works differently
Nothing in those logs (other than "Exiting Master process...") looks unusual. Can you get information on what actually killed the haproxy container from Docker inspect?
Not seeing much here, was there even an attempt to restart it?
[─ docker inspect 68c11dc186c1
{
"Id": "68c11dc186c167288e6b08e857af8887359ab4a81fa943dc8bbc1ebd59709946",
"Created": "2022-12-20T13:45:28.762773705Z",
"Path": "haproxy",
"Args": [
"-sf",
"7",
"-W",
"-db",
"-f",
"/usr/local/etc/haproxy/haproxy.cfg"
],
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2022-12-20T13:45:29.042208224Z",
"FinishedAt": "2022-12-20T13:52:04.616036701Z"
},
"Image": "sha256:083ad526a17e665e04888ac20ffb085a3a83b058b831ea5cf28164c9fc36c306",
"ResolvConfPath": "/var/lib/docker/containers/68c11dc186c167288e6b08e857af8887359ab4a81fa943dc8bbc1ebd59709946/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/68c11dc186c167288e6b08e857af8887359ab4a81fa943dc8bbc1ebd59709946/hostname",
"HostsPath": "/var/lib/docker/containers/68c11dc186c167288e6b08e857af8887359ab4a81fa943dc8bbc1ebd59709946/hosts",
"LogPath": "/var/lib/docker/containers/68c11dc186c167288e6b08e857af8887359ab4a81fa943dc8bbc1ebd59709946/68c11dc186c167288e6b08e857af8887359ab4a81fa943dc8bbc1ebd59709946-json.log",
"Name": "/ecoqube-dev-lb",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "unconfined",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "kind",
"PortBindings": {
"6443/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "37973"
}
]
},
"RestartPolicy": {
"Name": "unless-stopped",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/df2e414347b2c3b8a77ec5fb11fc5136305a5e84314c84da268c0e3fda126082-init/diff:/var/lib/docker/overlay2/58b22042a6cda25a15ca6e1c3a39500b2c3b24bc0a5fa34c7b11058346396628/diff:/var/lib/docker/overlay2/1ea3071304fa913301959e1da221f569e18505520d99078de07aea0a2d84f51a/diff:/var/lib/docker/overlay2/f5b35a9600d3b7350e48b753302a7c0d5a3bae7b48bd637785046546acb57d23/diff:/var/lib/docker/overlay2/cd9ece03c61a95227cd7df10670ad59882e0c0ecbee17b5a3a9de5ae2281f015/diff:/var/lib/docker/overlay2/9d3af073ed4440b6e4c94d17ca0479f25817fb19edae3e2c55e70b7b5a275596/diff",
"MergedDir": "/var/lib/docker/overlay2/df2e414347b2c3b8a77ec5fb11fc5136305a5e84314c84da268c0e3fda126082/merged",
"UpperDir": "/var/lib/docker/overlay2/df2e414347b2c3b8a77ec5fb11fc5136305a5e84314c84da268c0e3fda126082/diff",
"WorkDir": "/var/lib/docker/overlay2/df2e414347b2c3b8a77ec5fb11fc5136305a5e84314c84da268c0e3fda126082/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "bdb23fbb33fdde3878015b72fe4456da150804f1a44e320a9cd23e36bcfae1eb",
"Source": "/var/lib/docker/volumes/bdb23fbb33fdde3878015b72fe4456da150804f1a44e320a9cd23e36bcfae1eb/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "ecoqube-dev-lb",
"Domainname": "",
"User": "0",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"37973/tcp": {},
"6443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"
],
"Cmd": null,
"Image": "kindest/haproxy:v20210715-a6da3463",
"Volumes": {
"/var": {}
},
"WorkingDir": "/",
"Entrypoint": [
"haproxy",
"-sf",
"7",
"-W",
"-db",
"-f",
"/usr/local/etc/haproxy/haproxy.cfg"
],
"OnBuild": null,
"Labels": {
"io.x-k8s.kind.cluster": "ecoqube-dev",
"io.x-k8s.kind.role": "external-load-balancer"
},
"StopSignal": "SIGUSR1"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "4f91480bf67346919d74edd49617018fd92167f1876387945ea67630f99bab86",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/4f91480bf673",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"kind": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"68c11dc186c1",
"ecoqube-dev-lb"
],
"NetworkID": "29a148be6436fea9f8a0d3e5eb550756fcf513d75c01f7d15b58135e3f6ab785",
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"DriverOpts": null
}
}
}
}
]
Hmm - I'm really not sure what's going on here - maybe there's some additional info in journalctl for docker? I still don't really understand what causes the haproxy container to exit.
Just for confirmation. The current error is still:
E1213 13:40:20.163324 1 controller.go:326] "Reconciler error" err="failed to get ip for the load balancer: load balancer IP cannot be empty: container capi-quickstart-lb does not have an associated IP address" controller="dockercluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="DockerCluster" DockerCluster="default/capi-quickstart-lmhsm" namespace="default" name="capi-quickstart-lmhsm" reconcileID=fb7ba986-d9c7-4597-8f8a-b7d989a01601
The last time I had this error my docker network was not configured correctly (the docker network didn't exist) and thus the lb container didn't get an IP. Usually kind create
creates the network in the Docker engine and then CAPD re-uses it. A correct network should look something like this:
$ docker network list -f name=kind
NETWORK ID NAME DRIVER SCOPE
56aa1f164407 kind bridge local
$ docker network inspect kind
[
{
"Name": "kind",
"Id": "56aa1f164407508286bd7e7b56f48dc40c2b2f3be4566abed9f50e2c60d8ec03",
"Created": "2022-10-20T20:04:57.24810756+02:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": true,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1"
},
{
"Subnet": "fc00:f853:ccd:e793::/64",
"Gateway": "fc00:f853:ccd:e793::1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"a919a4b491f5ca226117ddb39667374c8c53bd7e5394d75c30ead489222973d0": {
"Name": "capi-test-control-plane",
"EndpointID": "791a6236f4dbe9e1bb68e7a7354bc8315caf15d5f5100c1f763b0d1b8622cfca",
"MacAddress": "02:42:ac:12:00:02",
"IPv4Address": "172.18.0.2/16",
"IPv6Address": "fc00:f853:ccd:e793::2/64"
},
"e46becf30116d3f0f2755fbb02c510e5f1ba7f540892108c214e34fe7840968a": {
"Name": "kind-registry",
"EndpointID": "89b3cffdbc97b929dea46e96f1e5987975a2f983729a424e5618638e1bd131b3",
"MacAddress": "02:42:ac:12:00:06",
"IPv4Address": "172.18.0.6/16",
"IPv6Address": "fc00:f853:ccd:e793::6/64"
}
},
"Options": {
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
]
(IPv6 stuff can vary a bit based on config)
Hi sbueringer I hope you had great holidays. This is what I get by running your same commands:
$ docker network list -f name=kind
NETWORK ID NAME DRIVER SCOPE
29a148be6436 kind bridge local
$ docker network inspect kind
[
{
"Name": "kind",
"Id": "29a148be6436fea9f8a0d3e5eb550756fcf513d75c01f7d15b58135e3f6ab785",
"Created": "2022-05-06T10:19:41.805997131+02:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": true,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1"
},
{
"Subnet": "fc00:f853:ccd:e793::/64",
"Gateway": "fc00:f853:ccd:e793::1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"14ee3d9ceecd3b62014a516b40c03fa681d8d59e7b5967088537d16e4b4932c6": {
"Name": "ecoqube-dev-cjltn-9fnfn",
"EndpointID": "aa3a717ee19355379050f6f67282fcf0827310f28eb4b6b060f6725ff0337529",
"MacAddress": "02:42:ac:12:00:02",
"IPv4Address": "172.18.0.2/16",
"IPv6Address": "fc00:f853:ccd:e793::2/64"
},
"34fb562bdbb4e61076c202e5ede80a01e39f93d3451a522421c27885d140385f": {
"Name": "ecoqube-dev-default-worker-topo-dt48g-b99d74ddc-nd6dg",
"EndpointID": "5b176229c224cb0f50d49188eb9cb302cf97b2f330f34ac71398414e4aced489",
"MacAddress": "02:42:ac:12:00:05",
"IPv4Address": "172.18.0.5/16",
"IPv6Address": "fc00:f853:ccd:e793::5/64"
},
"46d9d4c9fd27ea21d43ff7e547ecf36b8c1daaeab70828a62fb1692ed7452e80": {
"Name": "kind-control-plane",
"EndpointID": "923463d891f6270906fac0696707cc3de9a85c6cda8099469beea22322c51a66",
"MacAddress": "02:42:ac:12:00:04",
"IPv4Address": "172.18.0.4/16",
"IPv6Address": "fc00:f853:ccd:e793::4/64"
},
"eab855ab9c9a715a2a07ee6c4ab55b1a1b0df4bdeb0e742b17ccf308c63fc7eb": {
"Name": "ecoqube-dev-default-worker-topo-dt48g-b99d74ddc-vhjkd",
"EndpointID": "23ec82c914495d062d9ea717d47393fb1508bc7438ff24159f56f5f617291bac",
"MacAddress": "02:42:ac:12:00:06",
"IPv4Address": "172.18.0.6/16",
"IPv6Address": "fc00:f853:ccd:e793::6/64"
},
"fea8e436fb47f9488c685d2d7d487f91e77e544e64c8ce800f9a163436683b97": {
"Name": "ecoqube-dev-default-worker-topo-dt48g-b99d74ddc-mxj5x",
"EndpointID": "32f99c00929cf40d6ffec41cef7b45d2c40e0d6c9e86858ced4c25a4104f335d",
"MacAddress": "02:42:ac:12:00:03",
"IPv4Address": "172.18.0.3/16",
"IPv6Address": "fc00:f853:ccd:e793::3/64"
}
},
"Options": {
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
]
the proxy is not present. I wonder if I should seek to solve this somewhere more Docker related than the CAPI repo.
/triage not-reproducible
I wonder if I should seek to solve this somewhere more Docker related than the CAPI repo.
I kind of agree, it seems there is something going on at a lower level than the one where CAPI works
Hm yup. Network looks okay.
Sorry to necro this issue, but I think that I've found a reasonable way to reproduce this problem.
This has been happening to me when I have an old version of the haproxy container that has been killed in docker, but not removed. This typically happens when I lose my kind cluster that I run CAPI inside of and can no longer delete the cluster.
In this case, running docker ps -a
on the machine will show the old containers:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2e95e78b5c7c kindest/node:v1.26.3 "/usr/local/bin/entr…" 17 minutes ago Up 17 minutes 127.0.0.1:46021->6443/tcp kubemark-control-plane
7da7c7cb20ef kindest/node:v1.26.3 "/usr/local/bin/entr…" 17 minutes ago Up 17 minutes kubemark-worker
581d77fac175 kindest/node:v1.25.3 "/usr/local/bin/entr…" 3 weeks ago Exited (130) 2 weeks ago kube-node-mgmt-control-plane-gxc9k
8a1aca619a83 kindest/haproxy:v20230227-d46f45b6 "haproxy -sf 7 -W -d…" 3 weeks ago Exited (0) 2 weeks ago kube-node-mgmt-lb
Using docker rm <container_id>
at this point, will make it so that CAPI will come up correctly.
Steps for full reproduction:
kind create cluster ...
)clusterctl init ...
)clusterctl generate cluster ... | kubectl apply -f -
)kind delete clusters ...
)docker kill <container_id>
) - This is just for simulation, in every day course I find that they usually kill themselves after a certain amount of time, or die and are never restarted2023-05-10T01:39:50.582661780Z E0510 01:39:50.582602 1 controller.go:329] "Reconciler error" err="failed to get ip for the load balancer: load balancer IP cannot be empty: container kube-node-mgmt-lb does not have an associated IP address" controller="dockercluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="DockerCluster" DockerCluster="default/kube-node-mgmt" namespace="default" name="kube-node-mgmt" reconcileID=77d50330-7185-4279-9891-7bbc5696de96
docker rm <container_id>
kubectl delete cluster ...
It seems that if there is a remnant of the haproxy container hanging around, CAPD will wait around forever for it to come back up rather than either starting it again, or removing the container and re-creating it.
I don't know a ton about this codebase, but from what I can see, the loadbalancer container gets looked up here: https://github.com/kubernetes-sigs/cluster-api/blob/main/test/infrastructure/docker/internal/docker/loadbalancer.go#L62
And when the container exists, but is not running, then container
is set on the LoadBalancer
.
Later on when we then attempt to try to create the load balancer, we do a check to see if the container already exists (https://github.com/kubernetes-sigs/cluster-api/blob/main/test/infrastructure/docker/internal/docker/loadbalancer.go#L119) and since it does, we skip creating it.
However, at this point it is shut down and doesn't have an IP address on it so then when we go to fetch the IP later in the logic (https://github.com/kubernetes-sigs/cluster-api/blob/main/test/infrastructure/docker/internal/docker/loadbalancer.go#L201) we fall into the error condition.
@criscola would you be willing to consider re-opening this issue and look into the code points above to see if there is some way that this condition could be handled better?
What steps did you take and what happened: I create a CAPD cluster locally following the quickstart. Complete list of steps:
then I look at the CAPD controller's logs, e.g.
where I see the logs:
on further inspection with
docker ps
, the Docker container capi-quickstart-lb is, as expected, not present.At this point I delete the cluster
kubectl delete cluster capi-quickstart
and recreate it again. Immediately after, the loadbalancer Docker container gets created and the control plane results initialized. Weird, right?What did you expect to happen: I expect my cluster to not require recreation to become initialized.
Environment:
kubectl version
):uname -a
):Server: Engine: Version: 20.10.21 API version: 1.41 (minimum version 1.12) Go version: go1.19.2 Git commit: 3056208812 Built: Thu Oct 27 21:29:34 2022 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.6.9 GitCommit: 1c90a442489720eec95342e1789ee8a5e1b9536f.m runc: Version: 1.1.4 GitCommit:
docker-init: Version: 0.19.0 GitCommit: de40ad0