cloud-provider-kind container unable to connect to kind node IP address

[uromahn]

Environment:

• Apple Mac with Apple Silicon M1 Pro running Mac OS Sonoma 14.7.1.
• Rancher Desktop 1.16.0 running dockerd (moby).

  
  docker version
  Client:
  Version:           27.2.1-rd
  API version:       1.45 (downgraded from 1.47)
  Go version:        go1.22.7
  Git commit:        cc0ee3e
  Built:             Tue Sep 10 15:41:09 2024
  OS/Arch:           darwin/arm64
  Context:           default

Server: Engine: Version: 26.1.5 API version: 1.45 (minimum version 1.24) Go version: go1.22.5 Git commit: 411e817ddf710ff8e08fa193da80cb78af708191 Built: Fri Jul 26 17:51:06 2024 OS/Arch: linux/arm64 Experimental: false containerd: Version: v1.7.17 GitCommit: 3a4de459a68952ffb703bbe7f2290861a75b6b67 runc: Version: 1.1.14 GitCommit: 2c9f5602f0ba3d9da1c2596322dfc4e156844890 docker-init: Version: 0.19.0 GitCommit:

kind version kind v0.24.0 go1.22.6 darwin/arm64

I was following the steps to install a single node kind cluster using K8s 1.29.8 and started `cloud-provider-kind` with `sudo` in another terminal window.
I then installed the simple `foo-bar` app using:

kubectl apply -f https://kind.sigs.k8s.io/examples/loadbalancer/usage.yaml

After a while the cloud-provider-kind logs showed that the load balancer had been initialized and the lb IP was reported as

kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE foo-service LoadBalancer 10.96.102.151 172.19.0.3 5678:31933/TCP 12m

From my host machine, I am able to `ping 172.19.0.3` but when running `curl` against the lb IP, I am getting the following:

curl -v http://172.19.0.3:5678

• Trying 172.19.0.3:5678...
• Connected to 172.19.0.3 (172.19.0.3) port 5678

  GET / HTTP/1.1 Host: 172.19.0.3:5678 User-Agent: curl/8.7.1 Accept: /

• Request completely sent off

  and `curl` hangs.
  Looking at the Envoy access logs I then see the following:

  [2024-11-19T05:46:44.844Z] "- - -" 0 UF,URX 0 0 1352 - "-" "-" "-" "-" "172.19.0.2:31933"

I then exec into the kindccm-XXXX container and try to ping the kind control plane IP after installing ping, I am getting host unreachable. However, when running a plain Ubuntu container attached to the kind network, I am able to ping the control plane IP from there. When exec into the kind control plane container and trying to ping the IP address of the kindccm container, I am getting Destination Port Unreachable. I am able to ping the Ubuntu container from the kindccm container. However, when i curl the Envoy admin port 10000 from the host, I am able to reach it.

There is nothing unusual noticable in the IP route and no obvious difference between the routing table of the Ubuntu and the kindccm container.

Conclusion: for some unknown reason, connectivity between the kindccm container and the kind control plane container is blocked.

[aojea]

are the kind loadbalancers and the kind nodes in the same network?

I think with docker inspect for each container and docker network we can figure out these details