search

kubernetes-sigs / kind

Kubernetes IN Docker - local clusters for testing Kubernetes
https://kind.sigs.k8s.io/
Apache License 2.0
13.5k stars 1.56k forks source link

Kind fails creating cluster with Podman using XMARK in ip6 table #3765

Closed metal3d closed 2 weeks ago

metal3d commented 2 weeks ago

What happened: I only use Podman. I start a simple cluster with kind create cluster.

An error appears:

enabling experimental podman provider
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.31.0) 🖼
 ✗ Preparing nodes 📦
Deleted nodes: ["kind-control-plane"]
ERROR: failed to create cluster: command "podman run --name kind-control-plane --hostname kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --tmpfs /tmp --tmpfs /run --volume 167b013be7b74a504e45d7ad025350067d03e299806121e21af4e072b5d1f51c:/var:suid,exec,dev --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --net kind --label io.x-k8s.kind.cluster=kind -e container=podman --cgroupns=private --volume /dev/mapper:/dev/mapper --device /dev/fuse --publish=127.0.0.1:33667:6443/tcp -e KUBECONFIG=/etc/kubernetes/admin.conf docker.io/kindest/node@sha256:53df588e04085fd41ae12de0c3fe4c72f7013bba32a20e7325357a1ac94ba865" failed with error: exit status 126
Command Output: Error: netavark: unable to append rule '-j MARK  --set-xmark 0x2000/0x2000' to table 'nat': code: 2, msg: Warning: Extension MARK revision 0 not supported, missing kernel module?
ip6tables v1.8.10 (nf_tables): unknown option "--set-xmark"
Try `ip6tables -h' or 'ip6tables --help' for more information.

What you expected to happen:

The cluster should be up without any error 😄

How to reproduce it (as minimally and precisely as possible):

Using Fedora 40, podman 5.2.3, the error is present.

Anything else we need to know?:

Yes, there is workaround: create the "kind" network myself:

kind delete cluster
# drop the network created by "kind"
podman network rm kind

# create the network
podman network create kind

# and start the cluster
kind create cluster

And then, the cluster starts without any error, and without any problem.

Environment:

aojea commented 2 weeks ago

Error: netavark: unable to append rule '-j MARK --set-xmark 0x2000/0x2000' to table 'nat': code: 2, msg: Warning: Extension MARK revision 0 not supported, missing kernel module? ip6tables v1.8.10 (nf_tables): unknown option "--set-xmark"

you need that kernel module loaded to avoid this problem ... is a problem of the podman implementation of networks and your particular OS configuration, nothing kind can do I'm afraid

Try to load the corresponding module

sudo modprobe xt_mark
metal3d commented 2 weeks ago

Yes, but this module is not present in Fedora, however, recent kernel and generally Fedora is up to date...

metal3d commented 2 weeks ago

Ho... wait... actually the "modprobe" completion doesn't show it, but I can load it... let me make some tests

metal3d commented 2 weeks ago

No, doesn't work:

$ lsmod | grep xt_mark
xt_mark                12288  0

$ kind delete cluster
enabling experimental podman provider
Deleting cluster "kind" ...

$ podman network rm kind
kind

$ kind create cluster
enabling experimental podman provider
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.31.0) 🖼
 ✗ Preparing nodes 📦
Deleted nodes: ["kind-control-plane"]
ERROR: failed to create cluster: command "podman run --name kind-control-plane --hostname kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --tmpfs /tmp --tmpfs /run --volume b8218fbeb32376fcea0e90df379c12b53260ead2bd182fad649e122d57b967f3:/var:suid,exec,dev --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --net kind --label io.x-k8s.kind.cluster=kind -e container=podman --cgroupns=private --volume /dev/mapper:/dev/mapper --device /dev/fuse --publish=127.0.0.1:39289:6443/tcp -e KUBECONFIG=/etc/kubernetes/admin.conf docker.io/kindest/node@sha256:53df588e04085fd41ae12de0c3fe4c72f7013bba32a20e7325357a1ac94ba865" failed with error: exit status 126
Command Output: Error: netavark: unable to append rule '-j MARK  --set-xmark 0x2000/0x2000' to table 'nat': code: 2, msg: Warning: Extension MARK revision 0 not supported, missing kernel module?
ip6tables v1.8.10 (nf_tables): unknown option "--set-xmark"
Try `ip6tables -h' or 'ip6tables --help' for more information.
metal3d commented 2 weeks ago

Jus in case of, here is the network created by "kind" command:

[
     {
          "name": "kind",
          "id": "0a5e5508fc0854972d8389af663c33ec25a8eb891c52152b33a8f733be298440",
          "driver": "bridge",
          "network_interface": "podman5",
          "created": "2024-10-25T10:30:31.43505612+02:00",
          "subnets": [
               {
                    "subnet": "fc00:f853:ccd:e793::/64",
                    "gateway": "fc00:f853:ccd:e793::1"
               },
               {
                    "subnet": "10.89.4.0/24",
                    "gateway": "10.89.4.1"
               }
          ],
          "ipv6_enabled": true,
          "internal": false,
          "dns_enabled": true,
          "ipam_options": {
               "driver": "host-local"
          },
          "containers": {}
     }
]

And the one created manyally:

[
     {
          "name": "kind",
          "id": "a7f61d4f9239270fcd5e59e15911333fd77f8b0a8b1f17e1499dc6c0f6708aaa",
          "driver": "bridge",
          "network_interface": "podman5",
          "created": "2024-10-25T10:35:29.751360901+02:00",
          "subnets": [
               {
                    "subnet": "10.89.4.0/24",
                    "gateway": "10.89.4.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "ipam_options": {
               "driver": "host-local"
          },
          "containers": {}
     }
]

It means that the problem resides with ipv6 only I guess.

metal3d commented 2 weeks ago

OK, if I create the network with --ipv6, kind fails to start the cluster.

So, the problem is the MARK option on ip6tables.

aojea commented 2 weeks ago

OK, if I create the network with --ipv6, kind fails to start the cluster.

So, the problem is the MARK option on ip6tables.

can you report this to podman maintainers?

it seems they have this dependency on netavark, that is IIRC how podman implements the containers networks

Command Output: Error: netavark: unable to append rule '-j MARK --set-xmark 0x2000/0x2000' to table 'nat': code: 2, msg: Warning: Extension MARK revision 0 not supported, missing kernel module? ip6tables v1.8.10 (nf_tables): unknown option "--set-xmark"

metal3d commented 2 weeks ago

Actually it seems that Fedora has no "xt_CONNMARK" module (this is where the --set-xmark option is managed, looking inside the source here: https://www.netfilter.org/pub/iptables/iptables-1.4.1-rc2/extensions/libxt_CONNMARK.c).

I will check if it's not to Fedora project I need to report this. For now, creating the network without ipv6 is the workaround that make it working.

aojea commented 2 weeks ago

Thanks for the investigation and explanation

metal3d commented 2 weeks ago

Issue filled in Fedora BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2321716

Thanks for your help on investigation :)

metal3d commented 2 weeks ago

That was reported a few times before actually: https://bugzilla.redhat.com/show_bug.cgi?id=2321325

Bug will be resolved in a while

grooverdan commented 1 week ago

https://bugzilla.redhat.com/show_bug.cgi?id=2321325#c12 this work around worked for me on f40.