search

containers / netavark

Container network stack
Apache License 2.0
534 stars 84 forks source link

Rootfull netavark with nftables: internal:0:0-0: Error: Could not process rule: No such file or directory #1057

Closed mluhmann closed 2 months ago

mluhmann commented 2 months ago

I am running a custom yocto build linux with podman and netavark. Until now I didn't change the firewall driver, so it was running with iptables and parsed via iptables-nft to run with nft. But the system ran into some issues with the firewall rules and I would like to avoid the parser to get rid of this factor.

I configured podman accodingly and do get the following debug output:

image

Here a "podman info" output for all the version information:

podman info
host:
  arch: arm
  buildahVersion: 1.35.3
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: affab49967eb62f75d2a47398344ab053326289f'
  cpuUtilization:
    idlePercent: 76.72
    systemPercent: 9.43
    userPercent: 13.85
  cpus: 2
  databaseBackend: sqlite
  distribution:
    codename: scarthgap
    distribution: pxc
    version: 9999.0.0
  eventLogger: journald
  freeLocks: 2047
  hostname: axcf2152
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.1.33-rt11-pxc
  linkmode: dynamic
  logDriver: journald
  memFree: 82440192
  memTotal: 512811008
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.11.0-dev
    package: Unknown
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: Unknown
    path: /bin/crun
    version: |-
      crun version 1.14.3.0.0.0.8-89d4-dirty
      commit: 89d44467e3b410b73f2065756a12789be45b855b
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: Unknown
    version: |
      pasta 2024_05_23.765eb0b
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.0-beta.0+dev
      commit: unknown
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 0h 22m 38.00s
  variant: v7
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  '*.docker.io':
    Blocked: false
    Insecure: false
    Location: ""
    MirrorByDigestOnly: false
    Mirrors:
    - Insecure: false
      Location: mirror.gcr.io
      PullFromMirror: ""
    - Insecure: false
      Location: quay.io/libpod
      PullFromMirror: ""
    Prefix: '*.docker.io'
    PullFromMirror: ""
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:5000
    PullFromMirror: ""
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 7688126464
  graphRootUsed: 419758080
  graphStatus:
    Backing Filesystem: overlayfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /opt/system/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.0.2-dev
  Built: 1711987427
  BuiltTime: Mon Apr  1 16:03:47 2024
  GitCommit: bb81e85a430fa95d23a15b77c717fd68bf06ebf2
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/arm
  Version: 5.0.2-dev
Luap99 commented 2 months ago

Do you have the nftables modules installed/enable din the kernel? Can you run basic nft commands to add a table/chains/rules? This sounds like a env issue not something wrong with netavark.

Luap99 commented 2 months ago

compare: https://github.com/containers/netavark/issues/1054

mluhmann commented 2 months ago

nft is running fine. My system is writing rules automatically when the firewall is activated and the user can create rules via a webinterface. Additionally, iptables-nft is working fine. Here an output when podman/netavark is configured to use iptables:

# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
        chain NETAVARK-89B37C96258AB {
                ip daddr 10.89.0.0/24 counter accept
                ip daddr != 224.0.0.0/4 counter xt target "MASQUERADE"
        }

        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
                counter jump NETAVARK-HOSTPORT-MASQ
                ip saddr 10.89.0.0/24 counter jump NETAVARK-89B37C96258AB
        }

        chain NETAVARK-HOSTPORT-SETMARK {
                counter xt target "MARK"
        }

        chain NETAVARK-HOSTPORT-MASQ {
                xt match "comment" meta mark & 0x00002000 == 0x00002000 counter xt target "MASQUERADE"
        }

        chain NETAVARK-DN-89B37C96258AB {
                ip saddr 10.89.0.0/24 tcp dport 8080 counter jump NETAVARK-HOSTPORT-SETMARK
                ip saddr 127.0.0.1 tcp dport 8080 counter jump NETAVARK-HOSTPORT-SETMARK
                tcp dport 8080 counter xt target "DNAT"
        }

        chain NETAVARK-HOSTPORT-DNAT {
                tcp dport 8080 xt match "comment" counter jump NETAVARK-DN-89B37C96258AB
        }

        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                xt match "addrtype" counter jump NETAVARK-HOSTPORT-DNAT
        }

        chain OUTPUT {
                type nat hook output priority dstnat; policy accept;
                xt match "addrtype" counter jump NETAVARK-HOSTPORT-DNAT
        }
}
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
        chain NETAVARK_ISOLATION_2 {
        }

        chain NETAVARK_ISOLATION_3 {
                oifname "podman1" counter drop
                counter jump NETAVARK_ISOLATION_2
        }

        chain NETAVARK_INPUT {
                ip saddr 10.89.0.0/24 udp dport 53 counter accept
        }

        chain NETAVARK_FORWARD {
                xt match "conntrack" counter drop
                ip daddr 10.89.0.0/24 xt match "conntrack" counter accept
                ip saddr 10.89.0.0/24 counter accept
        }

        chain FORWARD {
                type filter hook forward priority filter; policy accept;
                xt match "comment" counter jump NETAVARK_FORWARD
        }

        chain INPUT {
                type filter hook input priority filter; policy accept;
                xt match "comment" counter jump NETAVARK_INPUT
        }
}
Luap99 commented 2 months ago

Try to run strace -f on the podman command to see where it fails in nft. I don't think we do any special rules that depend on newer versions but maybe your nft and/or kernel is to old? I don't think we tested on older versions than nftables v1.0.7 (at least that is what is used in fedora 39).

It certainly doesn't help that the nftables error is just "could not process rule" without saying which rule, given we batch all rules in one nft command it is impossible to know this.

mluhmann commented 2 months ago

nft is version 1.0.9, one before the latest version. The linux is build on the latest yocto release branch and all should be on the newest versions available around June. Here is the stacktrace: strace.txt

Luap99 commented 2 months ago 
[pid  2721] sendmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[[{nlmsg_len=20, nlmsg_type=0x10 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST, nlmsg_seq=0, nlmsg_pid=0}, "\x00\x00\x00\x0a"], [{nlmsg_len=44, nlmsg_type=0xa00 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST, nlmsg_seq=1, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x08\x00\x02\x00\x00\x00\x00\x00"], [{nlmsg_len=88, nlmsg_type=0xa03 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0x400, nlmsg_seq=2, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x0a\x00\x03\x00\x49\x4e\x50\x55\x54\x00\x00\x00"...], [{nlmsg_len=88, nlmsg_type=0xa03 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0x400, nlmsg_seq=3, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x0c\x00\x03\x00\x46\x4f\x52\x57\x41\x52\x44\x00"...], [{nlmsg_len=88, nlmsg_type=0xa03 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0x400, nlmsg_seq=4, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x10\x00\x03\x00\x50\x4f\x53\x54\x52\x4f\x55\x54"...], [{nlmsg_len=88, nlmsg_type=0xa03 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0x400, nlmsg_seq=5, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x0f\x00\x03\x00\x50\x52\x45\x52\x4f\x55\x54\x49"...], [{nlmsg_len=84, nlmsg_type=0xa03 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0x400, nlmsg_seq=6, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x0b\x00\x03\x00\x4f\x55\x54\x50\x55\x54\x00\x00"...], [{nlmsg_len=64, nlmsg_type=0xa03 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0x400, nlmsg_seq=7, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x1b\x00\x03\x00\x4e\x45\x54\x41\x56\x41\x52\x4b"...], [{nlmsg_len=68, nlmsg_type=0xa03 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0x400, nlmsg_seq=8, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x1e\x00\x03\x00\x4e\x45\x54\x41\x56\x41\x52\x4b"...], [{nlmsg_len=64, nlmsg_type=0xa03 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0x400, nlmsg_seq=9, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x19\x00\x03\x00\x4e\x45\x54\x41\x56\x41\x52\x4b"...], [{nlmsg_len=64, nlmsg_type=0xa03 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0x400, nlmsg_seq=10, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x19\x00\x03\x00\x4e\x45\x54\x41\x56\x41\x52\x4b"...], [{nlmsg_len=64, nlmsg_type=0xa03 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0x400, nlmsg_seq=11, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x19\x00\x03\x00\x4e\x45\x54\x41\x56\x41\x52\x4b"...], [{nlmsg_len=232, nlmsg_type=0xa06 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0xc00, nlmsg_seq=12, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x10\x00\x02\x00\x50\x4f\x53\x54\x52\x4f\x55\x54"...], [{nlmsg_len=220, nlmsg_type=0xa06 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0xc00, nlmsg_seq=13, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x1e\x00\x02\x00\x4e\x45\x54\x41\x56\x41\x52\x4b"...], [{nlmsg_len=216, nlmsg_type=0xa06 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0xc00, nlmsg_seq=14, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x0f\x00\x02\x00\x50\x52\x45\x52\x4f\x55\x54\x49"...], [{nlmsg_len=212, nlmsg_type=0xa06 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0xc00, nlmsg_seq=15, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x0b\x00\x02\x00\x4f\x55\x54\x50\x55\x54\x00\x00"...], [{nlmsg_len=244, nlmsg_type=0xa06 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0xc00, nlmsg_seq=16, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x0c\x00\x02\x00\x46\x4f\x52\x57\x41\x52\x44\x00"...], [{nlmsg_len=128, nlmsg_type=0xa06 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0xc00, nlmsg_seq=17, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x0c\x00\x02\x00\x46\x4f\x52\x57\x41\x52\x44\x00"...], [{nlmsg_len=208, nlmsg_type=0xa06 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0x400, nlmsg_seq=18, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x19\x00\x02\x00\x4e\x45\x54\x41\x56\x41\x52\x4b"...], [{nlmsg_len=144, nlmsg_type=0xa06 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0xc00, nlmsg_seq=19, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x19\x00\x02\x00\x4e\x45\x54\x41\x56\x41\x52\x4b"...], [{nlmsg_len=68, nlmsg_type=0xa03 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0x400, nlmsg_seq=20, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x1f\x00\x03\x00\x6e\x76\x5f\x32\x66\x32\x35\x39"...], [{nlmsg_len=296, nlmsg_type=0xa06 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0xc00, nlmsg_seq=21, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x1f\x00\x02\x00\x6e\x76\x5f\x32\x66\x32\x35\x39"...], [{nlmsg_len=336, nlmsg_type=0xa06 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0xc00, nlmsg_seq=22, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x1f\x00\x02\x00\x6e\x76\x5f\x32\x66\x32\x35\x39"...], [{nlmsg_len=452, nlmsg_type=0xa06 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0xc00, nlmsg_seq=23, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x0a\x00\x02\x00\x49\x4e\x50\x55\x54\x00\x00\x00"...], [{nlmsg_len=420, nlmsg_type=0xa06 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0xc00, nlmsg_seq=24, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x0c\x00\x02\x00\x46\x4f\x52\x57\x41\x52\x44\x00"...], [{nlmsg_len=276, nlmsg_type=0xa06 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0xc00, nlmsg_seq=25, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x0c\x00\x02\x00\x46\x4f\x52\x57\x41\x52\x44\x00"...], [{nlmsg_len=312, nlmsg_type=0xa06 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0xc00, nlmsg_seq=26, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x10\x00\x02\x00\x50\x4f\x53\x54\x52\x4f\x55\x54"...], [{nlmsg_len=20, nlmsg_type=0x11 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST, nlmsg_seq=27, nlmsg_pid=0}, "\x00\x00\x00\x0a"]], iov_len=4608}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0 <unfinished ...>

and then as response

[pid  2721] <... recvmsg resumed>{msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{nlmsg_len=244, nlmsg_type=NLMSG_ERROR, nlmsg_flags=NLM_F_ACK_TLVS, nlmsg_seq=14, nlmsg_pid=2721}, [{error=-ENOENT, msg=[{nlmsg_len=216, nlmsg_type=0xa06 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0xc00, nlmsg_seq=14, nlmsg_pid=0}, "\x01\x00\x00\x00\x0d\x00\x01\x00\x6e\x65\x74\x61\x76\x61\x72\x6b\x00\x00\x00\x00\x0f\x00\x02\x00\x50\x52\x45\x52\x4f\x55\x54\x49"...]}, [{nla_len=8, nla_type=NLMSGERR_ATTR_OFFS}, 56]]], iov_len=69652}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 244

strace cannot decode the netfilter attributes and I don't understand them either. If ENOENT is all we get then I See really no way to debug this.

Luap99 commented 2 months ago

kernel: 6.1.33-rt11-pxc

You kernel seems a bit old so maybe try a newer kernel.

mluhmann commented 2 months ago

Maybe I am still missing some kernel modules: https://unix.stackexchange.com/questions/537912/nftables-rule-no-such-file-or-directory-error/618789 But I don't know the rules netavark is trying to use. Here is an output of all NF modules.

CONFIG_NF_CONNTRACK=m
CONFIG_NF_LOG_SYSLOG=m
CONFIG_NF_CONNTRACK_MARK=y
# CONFIG_NF_CONNTRACK_ZONES is not set
CONFIG_NF_CONNTRACK_PROCFS=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CONNTRACK_TIMEOUT=y
CONFIG_NF_CONNTRACK_TIMESTAMP=y
# CONFIG_NF_CONNTRACK_LABELS is not set
# CONFIG_NF_CT_PROTO_DCCP is not set
CONFIG_NF_CT_PROTO_GRE=y
CONFIG_NF_CT_PROTO_SCTP=y
CONFIG_NF_CT_PROTO_UDPLITE=y
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_BROADCAST=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
# CONFIG_NF_CONNTRACK_SNMP is not set
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
# CONFIG_NF_CT_NETLINK_TIMEOUT is not set
CONFIG_NF_NAT=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_SIP=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_REDIRECT=y
CONFIG_NF_NAT_MASQUERADE=y
CONFIG_NF_TABLES=y
CONFIG_NF_TABLES_INET=y
CONFIG_NF_TABLES_NETDEV=y
# CONFIG_NFT_NUMGEN is not set
CONFIG_NFT_CT=m
CONFIG_NFT_CONNLIMIT=m
CONFIG_NFT_LOG=m
CONFIG_NFT_LIMIT=m
CONFIG_NFT_MASQ=m
CONFIG_NFT_REDIR=m
CONFIG_NFT_NAT=m
CONFIG_NFT_TUNNEL=m
CONFIG_NFT_OBJREF=m
CONFIG_NFT_QUEUE=m
# CONFIG_NFT_QUOTA is not set
CONFIG_NFT_REJECT=m
CONFIG_NFT_REJECT_INET=m
CONFIG_NFT_COMPAT=m
CONFIG_NFT_HASH=m
# CONFIG_NFT_XFRM is not set
CONFIG_NFT_SOCKET=m
# CONFIG_NFT_OSF is not set
# CONFIG_NFT_TPROXY is not set
# CONFIG_NFT_SYNPROXY is not set
# CONFIG_NF_DUP_NETDEV is not set
# CONFIG_NFT_DUP_NETDEV is not set
# CONFIG_NFT_FWD_NETDEV is not set
# CONFIG_NFT_REJECT_NETDEV is not set
CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_SOCKET_IPV4=m
# CONFIG_NF_TPROXY_IPV4 is not set
CONFIG_NF_TABLES_IPV4=y
CONFIG_NFT_REJECT_IPV4=m
# CONFIG_NFT_DUP_IPV4 is not set
# CONFIG_NFT_FIB_IPV4 is not set
CONFIG_NF_TABLES_ARP=y
# CONFIG_NF_DUP_IPV4 is not set
# CONFIG_NF_LOG_ARP is not set
CONFIG_NF_LOG_IPV4=m
CONFIG_NF_REJECT_IPV4=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_NF_SOCKET_IPV6=m
# CONFIG_NF_TPROXY_IPV6 is not set
CONFIG_NF_TABLES_IPV6=y
CONFIG_NFT_REJECT_IPV6=m
# CONFIG_NFT_DUP_IPV6 is not set
# CONFIG_NFT_FIB_IPV6 is not set
# CONFIG_NF_DUP_IPV6 is not set
CONFIG_NF_REJECT_IPV6=m
CONFIG_NF_LOG_IPV6=m
CONFIG_NF_DEFRAG_IPV6=m
CONFIG_NF_TABLES_BRIDGE=m
# CONFIG_NFT_BRIDGE_META is not set
# CONFIG_NFT_BRIDGE_REJECT is not set
# CONFIG_NF_CONNTRACK_BRIDGE is not set
# CONFIG_NFC is not set
# CONFIG_NFTL is not set
CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not set
CONFIG_NFS_V4=y
# CONFIG_NFS_V4_1 is not set
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFS_DISABLE_UDP_SUPPORT=y
CONFIG_NFSD=m
# CONFIG_NFSD_V3_ACL is not set
CONFIG_NFSD_V4=y
# CONFIG_NFSD_BLOCKLAYOUT is not set
# CONFIG_NFSD_SCSILAYOUT is not set
# CONFIG_NFSD_FLEXFILELAYOUT is not set
CONFIG_NFS_COMMON=y
Luap99 commented 2 months ago 
table inet netavark {
    chain INPUT {
        type filter hook input priority filter; policy accept;
        ip saddr 10.88.0.0/16 udp dport 53 accept
    }

    chain FORWARD {
        type filter hook forward priority filter; policy accept;
        ct state invalid drop
        jump NETAVARK-ISOLATION-1
        ip daddr 10.88.0.0/16 ct state established,related accept
        ip saddr 10.88.0.0/16 accept
    }

    chain POSTROUTING {
        type nat hook postrouting priority srcnat; policy accept;
        meta mark & 0x00002000 == 0x00002000 masquerade
        ip saddr 10.88.0.0/16 jump nv_2f259bab_10_88_0_0_nm16
    }

    chain PREROUTING {
        type nat hook prerouting priority dstnat; policy accept;
        fib daddr type local jump NETAVARK-HOSTPORT-DNAT
    }

    chain OUTPUT {
        type nat hook output priority -100; policy accept;
        fib daddr type local jump NETAVARK-HOSTPORT-DNAT
    }

    chain NETAVARK-HOSTPORT-DNAT {
    }

    chain NETAVARK-HOSTPORT-SETMARK {
        meta mark set meta mark | 0x00002000
    }

    chain NETAVARK-ISOLATION-1 {
    }

    chain NETAVARK-ISOLATION-2 {
    }

    chain NETAVARK-ISOLATION-3 {
        oifname "podman0" drop
        jump NETAVARK-ISOLATION-2
    }

    chain nv_2f259bab_10_88_0_0_nm16 {
        ip daddr 10.88.0.0/16 accept
        ip daddr != 224.0.0.0/4 masquerade
    }
}

This is our ruleset for a normal container on the default bridge network, but I have no idea what kernel modules would be needed. For reference this is from a working fedora kernel

CONFIG_NFT_NUMGEN=m
CONFIG_NFT_CT=m
CONFIG_NFT_FLOW_OFFLOAD=m
CONFIG_NFT_CONNLIMIT=m
CONFIG_NFT_LOG=m
CONFIG_NFT_LIMIT=m
CONFIG_NFT_MASQ=m
CONFIG_NFT_REDIR=m
CONFIG_NFT_NAT=m
CONFIG_NFT_TUNNEL=m
CONFIG_NFT_QUEUE=m
CONFIG_NFT_QUOTA=m
CONFIG_NFT_REJECT=m
CONFIG_NFT_REJECT_INET=m
CONFIG_NFT_COMPAT=m
CONFIG_NFT_HASH=m
CONFIG_NFT_FIB=m
CONFIG_NFT_FIB_INET=m
CONFIG_NFT_XFRM=m
CONFIG_NFT_SOCKET=m
# CONFIG_NFT_OSF is not set
CONFIG_NFT_TPROXY=m
CONFIG_NFT_SYNPROXY=m
CONFIG_NFT_DUP_NETDEV=m
CONFIG_NFT_FWD_NETDEV=m
CONFIG_NFT_FIB_NETDEV=m
CONFIG_NFT_REJECT_NETDEV=m
CONFIG_NFT_REJECT_IPV4=m
CONFIG_NFT_DUP_IPV4=m
CONFIG_NFT_FIB_IPV4=m
CONFIG_NFT_COMPAT_ARP=m
CONFIG_NFT_REJECT_IPV6=m
CONFIG_NFT_DUP_IPV6=m
CONFIG_NFT_FIB_IPV6=m
CONFIG_NFT_BRIDGE_META=m
CONFIG_NFT_BRIDGE_REJECT=m
# CONFIG_NFTL is not set
hluaces commented 2 months ago

For what is worth: this was also happening to us on an environment with WSL and Fedora 40. Same podman and nft versions. Eventually we rolled back to Fedora 38 with podman-4.9.4 to make it work.

Seems podman 5 has a lot of networking changes (including deprecating CNI without a flag) so probably one of those was playing tricks on us, still TBD.

Luap99 commented 2 months ago

For what is worth: this was also happening to us on an environment with WSL and Fedora 40. Same podman and nft versions. Eventually we rolled back to Fedora 38 with podman-4.9.4 to make it work.

Did you explicitly configure the nftables driver? nftables default is added to the Fedora 41 rpm so on f40 it still defaults to iptables unless you manually switched to nftables so I do not see how this would be related, especially not to the podman version.

mluhmann commented 2 months ago

My next step will be to figure out missing kernel modules for the rules. Currently I do not know where to get this information. The official documentation from nftables does only say:

You may also hit this problem if you forgot to compile the module that enables this chain type in your Linux kernel.

But the modules for the base chain types are available, as far as was able to figure this out.

mluhmann commented 2 months ago

I figured out some modules which could be missing, but most of them are not even available for kernel 6.x. Is there a way to output the ruleset netavark tries to set (and failes) or a list of all possible chain types netavark could generate rules for?

Luap99 commented 2 months ago

@mluhmann see the https://github.com/containers/netavark/issues/1057#issuecomment-2286149984 above for the normal ruleset on the default bridge network.

mluhmann commented 2 months ago

Thanks, it worked. I probably added to many kernel modules, but for testing purposes its ok. My next issue is related to pasta as rootless user.