Ingress fails to start, operation not permitted in "events" - Podman

metal3d commented 3 days ago

What happened:

Using kind to start a cluster, then apply the ingress-controller installation. The pods fails to go in condition "ready" and the events says:

2m34s       Warning   FailedCreatePodSandBox   pod/ingress-nginx-controller-5f4f4d9787-mvfxz    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4b6b4e5f6b6a3254b089d8702873e5843a8bd5b9ae34619848e46191d89b28bf": plugin type="portmap" failed (add): unable to create chain CNI-HOSTPORT-SETMARK: failed to check iptables chain existence: running [/usr/sbin/iptables -t nat -S CNI-HOSTPORT-SETMARK 1 --wait]: exit status 3: modprobe: ERROR: could not insert 'ip_tables': Operation not permitted...

What you expected to happen:

As one week ago, having a local cluster and the ingress working 😄

How to reproduce it (as minimally and precisely as possible):

kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
  - containerPort: 443
    hostPort: 443
EOF

kubectl wait --for=condition=Ready nodes --all --timeout=60s

kubectl apply -f https://kind.sigs.k8s.io/examples/ingress/deploy-ingress-nginx.yaml

Anything else we need to know?:

One week ago, everything worked without any problem. I don't know what happened (upgrade?) but it's now crashing.

Environment:

kind version: (use kind version): kind v0.25.0 go1.23.3 linux/amd64
Runtime info: (use docker info, podman info or nerdctl info):

host:
  arch: amd64
  buildahVersion: 1.38.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-3.fc41.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 93.62
    systemPercent: 3.21
    userPercent: 3.18
  cpus: 12
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "41"
  eventLogger: journald
  freeLocks: 2046
  hostname: patrice-laptop-home
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.11.8-300.fc41.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 9846616064
  memTotal: 29261115392
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.13.1-1.fc41.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.13.1
    package: netavark-1.13.0-1.fc41.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.13.0
  ociRuntime:
    name: crun
    package: crun-1.18.1-1.fc41.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.18.1
      commit: c41f034fdbb9742c395085fc98459c94ad1f9aae
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20241030.gee7d0b6-1.fc41.x86_64
    version: |
      pasta 0^20241030.gee7d0b6-1.fc41.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 0h 10m 21.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/metal3d/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/metal3d/.local/share/containers/storage
  graphRootAllocated: 510405902336
  graphRootUsed: 357890756608
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/metal3d/.local/share/containers/storage/volumes
version:
  APIVersion: 5.3.0
  Built: 1731456000
  BuiltTime: Wed Nov 13 01:00:00 2024
  GitCommit: ""
  GoVersion: go1.23.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.3.0

OS (e.g. from /etc/os-release):

NAME="Fedora Linux"
VERSION="41 (Workstation Edition)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=41
VERSION_CODENAME=""
PLATFORM_ID="platform:f41"
PRETTY_NAME="Fedora Linux 41 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:41"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f41/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=41
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=41
SUPPORT_END=2025-12-15
VARIANT="Workstation Edition"
VARIANT_ID=workstation

Kubernetes version: (use kubectl version):

Client Version: v1.30.6
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.31.2

Any proxies or other special environment settings?: EDIT: I forget to say that it's a rootless start, and I allowed my user to open port from 80 net.ipv4.ip_unprivileged_port_start = 80

metal3d commented 3 days ago

Precision taken from logs:

Nov 22 10:07:06 kind-control-plane kubelet[723]: E1122 10:07:06.291937     723 log.go:32] "StopPodSandbox from runtime service failed" err=<
Nov 22 10:07:06 kind-control-plane kubelet[723]:         rpc error: code = Unknown desc = failed to destroy network for sandbox "41bab85940ca4be1d3ada6a2eb22b11d51418ff4cbed344b6a7baa90f683135d": plugin type="portmap" failed (delete): neither iptables nor ip6tables is usable, (iptables) running [/usr/sbin/iptables -t nat -S OUTPUT --wait]: exit status 3: modprobe: ERROR: could not insert 'ip_tables': Operation not permitted
Nov 22 10:07:06 kind-control-plane kubelet[723]:         iptables v1.8.9 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Nov 22 10:07:06 kind-control-plane kubelet[723]:         Perhaps iptables or your kernel needs to be upgraded.
Nov 22 10:07:06 kind-control-plane kubelet[723]:         , (ip6tables) running [/usr/sbin/ip6tables -t nat -S OUTPUT --wait]: exit status 3: modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted
Nov 22 10:07:06 kind-control-plane kubelet[723]:         ip6tables v1.8.9 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Nov 22 10:07:06 kind-control-plane kubelet[723]:         Perhaps ip6tables or your kernel needs to be upgraded.
Nov 22 10:07:06 kind-control-plane kubelet[723]:  > podSandboxID="41bab85940ca4be1d3ada6a2eb22b11d51418ff4cbed344b6a7baa90f683135d"

On Fedora 41, these modules are not loaded and not found

BenTheElder commented 3 days ago

This is rootless with pasta? Have you seen: https://kind.sigs.k8s.io/docs/user/rootless/

attempting to run kubernetes rootlessly is a bit more fiddly and has some additional requirements

metal3d commented 3 days ago

That worked until Tuesday... For any reason, that fails yesterday.

metal3d commented 3 days ago

I just tried to force the delegation in systemd, and unfortunately it fails again. 😢

BenTheElder commented 3 days ago

There is a separate note there about iptables

BenTheElder commented 3 days ago

(Also in this case iptables usage is from the standard CNI plugins project)

metal3d commented 3 days ago

With apologies, I think I had mistyped the commands to inject the iptables modules - and now I don't have that error anymore.

Fedora may have recently changed the default injected modules.

Now I have a "simple" problem: the control-plane node listens to ports 80 and 443, but I get a connection reset by peer error when I try to access a site (for example, the demo in the doc).

Note: I've been using kind for years, this is the first time I've had this kind of problem.

metal3d commented 3 days ago

(also, port-forwarding 8080 to 80 on the ingress-controller works, the ports that doesn't works are the one in extra port mapping in the configuration shown in my initial comment)

(And thank you for your help)

metal3d commented 3 days ago

OK, forget the last problem 😃

I made changes after the first issue (the iptables problem). Back to the original configuration is OK.

Sorry for the noise. I must now alert my teamates that Fedora changes some things that force us to load iptables modules and, possibly, force delegation.

One more time, thanks for all (for your help, and for "kind")

BenTheElder commented 3 days ago

Fedora may have recently changed the default injected modules.

So my very rough understanding is that iptables is being phased out in redhat distros ^1 (?), but shouldn't be totally gone yet.

We are in touch with SIG Network about this, today I confirmed with one of the TLs (@danwinship) that CNI plugins 1.6.0 switches to nftables but we may want to wait for at least 1.6.1 for some bug fixes. Some future kind release will adopt those for new node images.

At some point in the not too distant future the ecosystem will likely be on nftables broadly and then you may not need to load those modules, but currently they are still necessary.

Thanks for understanding and the thank you, and I'm glad you got it figured out 😅

metal3d commented 2 days ago

No problem.

One point: something that I probably missed is this https://kind.sigs.k8s.io/docs/user/configuration/#kube-proxy-mode

When I change to nftables, the same error happens (without loading modules), is there any reason that iptables is still necessary when I set kubeProxyMode: "nftables"?

danwinship commented 2 days ago

CNI plugins 1.6.0 doesn't "switch to" nftables, it just supports nftables. But everything will still default to using iptables unless you request nftables in the CNI config, or you don't have iptables installed.

danwinship commented 2 days ago

@metal3d that changes the kube-proxy config but the errors here were coming from the CNI portmap plugin.

metal3d commented 1 day ago

Thanks for all these explanation.

kubernetes-sigs / kind

Ingress fails to start, operation not permitted in "events" - Podman #3798