microk8s service not starting on dual stack debian - kubelite just binds to tcp6

[hnz101]

Summary

Fresh snap install of latest stable or edge of microk8s doesn't run and the service won't come up.

For some reason kubelite just starts listening to ipv6, but wants to use the api via ipv4 which doesn't work.

root@extrac:~# snap install microk8s --classic
microk8s (1.28/stable) v1.28.3 from Canonical✓ installed

root@extrac:~# netstat -tulpn | grep 16443
tcp6       0      0 :::16443                :::*                    LISTEN      6075/kubelite

root@extrac:~# nc -w 3 -vz 127.0.0.1 16443
nc: connect to 127.0.0.1 port 16443 (tcp) timed out: Operation now in progress
root@extrac:~# nc -w 3 -vz ::1 16443
Connection to ::1 16443 port [tcp/*] succeeded!

Error from log:
microk8s.daemon-kubelite[6075]: W1211 12:13:21.130605    6075 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.CSINode: Get "https://127.0.0.1:16443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 127.0.0.1:16443: i/o timeout

Besides this timeout it can't connect to kine.sock, even tough it is present:

microk8s.daemon-kubelite[76025]: W1209 11:40:29.101089   76025 logging.go:59] [core] [Channel #2 SubChannel #3] grpc: addrConn.createTransport failed to connect to {
microk8s.daemon-kubelite[76025]:   "Addr": "unix:///var/snap/microk8s/6089/var/kubernetes/backend/kine.sock:12379",
microk8s.daemon-kubelite[76025]:   "ServerName": "kine.sock",
microk8s.daemon-kubelite[76025]:   "Attributes": null,
microk8s.daemon-kubelite[76025]:   "BalancerAttributes": null,
microk8s.daemon-kubelite[76025]:   "Type": 0,
microk8s.daemon-kubelite[76025]:   "Metadata": null
microk8s.daemon-kubelite[76025]: }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/snap/microk8s/6089/var/kubernetes/backend/kine.sock:12379: connect: no such file or directory"

What Should Happen Instead?

The Service should come up.

Reproduction Steps

Fresh Install latest snap version of microk8s.

Introspection Report

microk8s inspect hangs at "Inspect kubernetes cluster" and doesn't produce an tarball.

Is there an option to force IPv4 only? I already tried setting --bind-address in /var/snap/microk8s/6089/args/kube-apiserver, but that didn't help.

What information can I deliver to help you?

[neoaggelos]

Hi @hnz101, thank you for reporting the issue.

This issue in 1.28.3 was related to some regression in the Kubernetes code around dualstack handling, and IIRC should be resolved in 1.28.4 onwards.

Kubernetes 1.28.4 is not yet out to 1.28/stable, can you test if the issue is resolved if you install microk8s from 1.28/candidate?

You could do this with:

# install 
$ sudo snap install microk8s --channel 1.28/candidate --classic
# or refresh, if already installed
$ sudo snap refresh microk8s --channel 1.28/candidate

For reference, this is the Kubernetes version at the moment:

$ sudo snap info microk8s | grep 1.28/
  1.28/stable:           v1.28.3  2023-11-12 (6089) 185MB classic
  1.28/candidate:        v1.28.4  2023-11-22 (6225) 185MB classic
  1.28/beta:             v1.28.4  2023-11-22 (6225) 185MB classic
  1.28/edge:             v1.28.4  2023-12-05 (6310) 185MB classic

[hnz101]

Hi @neoaggelos and thanks for your reply, after some days of digging around I just found what caused my problem.

I used an iptables POSTROUTING / MASQUERADE rule to forward some local ports and this caused the weird behavior.

iptables -t nat -A POSTROUTING -j MASQUERADE

I replaced it with an more specific rule and now microk8s 1.28.3 and 1.28.4 comes up without problems.

iptables -t nat -A POSTROUTING -p tcp --dport 12345 -d 100.20.30.40 -j SNAT --to-source 192.168.123.123

Thank You!