search

kubernetes-sigs / kind

Kubernetes IN Docker - local clusters for testing Kubernetes
https://kind.sigs.k8s.io/
Apache License 2.0
13.5k stars 1.56k forks source link

Investigate docker v27 ipv6 changes #3677

Open BenTheElder opened 4 months ago

BenTheElder commented 4 months ago

See:

https://docs.docker.com/engine/release-notes/27.0/#ipv6

https://kubernetes.slack.com/archives/CEKK1KTN2/p1719537867758879

https://github.com/kubernetes/test-infra/pull/32863#issuecomment-2201170650

We need to figure out how we want to handle these in Kubernetes's CI and in kind.

/assign @aojea

aojea commented 4 months ago

Did we see any issues yet?

we have this weird problem here https://github.com/kubernetes-sigs/kube-network-policies/pull/47#issuecomment-2201263789 , but I do not know if is related

BenTheElder commented 4 months ago

Did we see any issues yet?

Yes.

You can see that just creating the network now flakes in pull-kind-test job, since we updated to docker v27 in CI earlier: https://github.com/kubernetes/test-infra/pull/32863#issuecomment-2201272818

BenTheElder commented 4 months ago

Example failure: https://github.com/kubernetes-sigs/kind/pull/648#issuecomment-2201265959

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kind/648/pull-kind-test/1807905535295492096

A user also reported issues with their own environment and docker v27 in the slack link above.

BenTheElder commented 4 months ago

=== FAIL: pkg/cluster/internal/providers/docker TestIntegrationEnsureNetworkConcurrent (1.45s) network_integration_test.go:60: error creating network: command "docker network create -d=bridge -o com.docker.network.bridge.enable_ip_masquerade=true -o com.docker.network.driver.mtu=1500 --ipv6 --subnet fc00:3051:9942:af9f::/64 integration-test-ensure-kind-network" failed with error: exit status 1 network_integration_test.go:63: "Error response from daemon: Failed to Setup IP tables: Unable to enable NAT rule: (iptables failed: ip6tables --wait -t nat -I POSTROUTING -s fc00:3051:9942:af9f::/64 ! -o br-4e53c7863d0d -j MASQUERADE: modprobe: FATAL: Module ip6_tables not found in directory /lib/modules/5.15.0-1054-gke\nip6tables v1.8.9 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)\nPerhaps ip6tables or your kernel needs to be upgraded.\n (exit status 3))\n" network_integration_test.go:65:

aojea commented 4 months ago

Commented in https://github.com/kubernetes/test-infra/pull/32863#issuecomment-2202678906

BenTheElder commented 4 months ago

working on fix in: https://github.com/kubernetes/test-infra/pull/32890 https://github.com/kubernetes/test-infra/pull/32891 + triggered an early re-run of autobump job following those and then merging: https://github.com/kubernetes/test-infra/pull/32881

BenTheElder commented 4 months ago

OK, I think this is mitigated for Kubernetes CI now, after also https://github.com/kubernetes/test-infra/pull/32895

We should still look into the behavior changes more and probably clean up the dind iptables stuff.

BenTheElder commented 3 months ago

Another variant, possibly, https://kubernetes.slack.com/archives/CEKK1KTN2/p1723750263635729

BenTheElder commented 2 months ago

and https://kubernetes.slack.com/archives/CEKK1KTN2/p1725478369195179