podman run container with network option can't connect to internet

shenwii commented 10 months ago

Issue Description

When I run the container with podman specifying network, the container has no way to access the internet.

Steps to reproduce the issue

1.add some conf to sysctl

# cat >/etc/sysctl.d/podman.conf <<EOF
#podman
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1
net.ipv4.ip_unprivileged_port_start=0
EOF

2.apply it # sysctl -p 3.create podman network $ podman network create --subnet 192.168.10.0/24 --gateway 192.168.10.1 --subnet fd52:2a5a:747e:3acd::/64 --gateway fd52:2a5a:747e:3acd::1 testnet

Describe the results you received

1.when run container with testnet, it's NG

$ podman run -it --rm --network testnet docker.io/ytabuchi/net-client curl http://www.google.com
curl: (7) Failed to connect to www.google.com port 80 after 76421 ms: Connection refused

2.when run container with default podman network, it's also NG

$ podman run -it --rm --network podman docker.io/ytabuchi/net-client curl http://www.google.com
curl: (7) Failed to connect to www.google.com port 80 after 75974 ms: Connection refused

3.but when i run container without network option is OK

$ podman run -it --rm docker.io/ytabuchi/net-client curl http://www.google.com
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="ja"><head><meta content="&#19990;&#30028;&#20013;&#12398;&#12354;&#12425;&#12422;&#12427;&#24773;&#22577;&#12434;&#26908;&#32034;&#12377;&#12427;&#12383;&#12417;&#12398;&#12484;&#12540;&#12523;&#12434;&#25552;&#20379;&#12375;&#12390;&#12356;&#12414;&#12377;&#12290;&#12373;&#12414;&#12374;&#12414;&#12394;&#26908;&#32034;&#27231;&#33021;&#12434;&#27963;&#29992;&#12375;&#12390;&#12289;&#12362;&#25506;&#12375;&#12398;&#24773;&#22577;&#12434;&#35211;&#12388;&#12369;&#12390;&#12367;&#12384;&#12373;&#12356;&#12290;" name="description"><meta content="noodp" name="robots"><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><title>Google</title><script nonce="u6OlXsU7LcWnL8m8KoZHaQ">(function(){var _g={kEI:'ApCeZf3QOq_i2roP7YWuuAw',kEXPI:'0,798230,3,567235,206,4804,1132070,870537,327182,380772,16114,28684,23792,284,12028,4752,12835,4998,17075,38444,2872,2891,4140,7614,606,29843,825,30022,16105,230,20583,4,59617,27041,6633,7596,1,42157,2,39758,5679,1021,31121,4568,6259,23416,1248,33070,2,2,1,6960,19672,8155,23351,20506,6,1923,9779,12414,30045,3142,17056,20151,82,2932,50014,3030,15816,1804,7759,6047,7205,14258,5728,6085,1632,8844,5257657,2,226,70,1139,151,15,5992154,1210,2806666,31560,7443905,19269717,88207,1182080,16672,43887,3,1603,3,262,3,234,3,2121276,2585,23029351,12799,8408.................

Describe the results you expected

podman run container with network can't connect to internet

podman info output

$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux trixie/sid"
NAME="Debian GNU/Linux"
VERSION_CODENAME=trixie
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ podman --version
podman version 4.7.2

$ podman inspect testnet 
[
     {
          "name": "testnet",
          "id": "6576cbf44d65a299687f13f643b97d806776a0dbb15d007e180a154b0052057e",
          "driver": "bridge",
          "network_interface": "podman1",
          "created": "2024-01-08T01:22:10.587395534Z",
          "subnets": [
               {
                    "subnet": "192.168.10.0/24",
                    "gateway": "192.168.10.1"
               },
               {
                    "subnet": "fd52:2a5a:747e:3acd::/64",
                    "gateway": "fd52:2a5a:747e:3acd::1"
               }
          ],
          "ipv6_enabled": true,
          "internal": false,
          "dns_enabled": true,
          "ipam_options": {
               "driver": "host-local"
          }
     }
]

$ podman info
host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.6+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 91.03
    systemPercent: 2.74
    userPercent: 6.23
  cpus: 4
  databaseBackend: boltdb
  distribution:
    codename: trixie
    distribution: debian
    version: unknown
  eventLogger: journald
  freeLocks: 2040
  hostname: nas
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 3000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 3000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 6.5.0-5-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 169836544
  memTotal: 16492843008
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-4_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_1.12-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.12
      commit: ce429cb2e277d001c2179df1ac66a470f00802ae
      rundir: /run/user/3000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20231204.b86afe3-1_amd64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/3000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 19659448320
  swapTotal: 19999485952
  uptime: 59h 18m 17.00s (Approximately 2.46 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /media/raid/nas/pod/.config/containers/storage.conf
  containerStore:
    number: 8
    paused: 0
    running: 8
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /media/raid/nas/pod/.local/share/containers/storage
  graphRootAllocated: 3000592982016
  graphRootUsed: 1162341236736
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 41
  runRoot: /run/user/3000/containers
  transientStore: false
  volumePath: /media/raid/nas/pod/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.2
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.21.3
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.2

firewall is not enabled.
# nft list ruleset
nothing is output

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Debian testing

Additional information

No response

Luap99 commented 10 months ago

Check podman unshare --rootless-netns ip addr, you should see a tap0 interface there, also check if slirp4netns is running. Did you try to reboot? Or at least stop all containers then rm $XDG_RUNTIME_DIR/netns/rootless-netns-*

shenwii commented 10 months ago

@Luap99

$ podman unshare --rootless-netns ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo 
       valid_lft forever preferred_lft forever
2: tap0: <BROADCAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 1e:a8:37:6a:8b:83 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.100/24 brd 10.0.2.255 scope global tap0
       valid_lft forever preferred_lft forever
    inet6 fe80::1ca8:37ff:fe6a:8b83/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: podman1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:58:a2:6c:c9:51 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.1/24 brd 192.168.10.255 scope global podman1
       valid_lft forever preferred_lft forever
    inet6 fd52:2a5a:747e:3acd::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::402e:38ff:fe05:4a24/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
18: veth0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master podman1 state UP group default qlen 1000
    link/ether 0a:58:a2:6c:c9:51 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::858:a2ff:fe6c:c951/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
19: veth3@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master podman1 state UP group default qlen 1000
    link/ether 62:f3:ae:83:a3:53 brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::60f3:aeff:fe83:a353/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
20: veth4@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master podman1 state UP group default qlen 1000
    link/ether f2:90:f9:f8:7b:d3 brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::f090:f9ff:fef8:7bd3/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
21: veth1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master podman1 state UP group default qlen 1000
    link/ether c2:a7:85:17:65:dc brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::c0a7:85ff:fe17:65dc/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
22: veth2@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master podman1 state UP group default qlen 1000
    link/ether 86:56:a2:9c:b6:eb brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::8456:a2ff:fe9c:b6eb/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
23: veth5@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master podman1 state UP group default qlen 1000
    link/ether e2:cd:1f:42:2a:a0 brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet6 fe80::e0cd:1fff:fe42:2aa0/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

tap0 interface is exists.

$ ps -ef | grep slirp4netns
pod         2317       1  0 Jan08 ?        00:00:17 /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -r 3 --netns-type=path /run/user/3000/netns/rootless-netns-d47816ac6169a5f68d2a tap0
pod       355666  354980  0 13:56 pts/0    00:00:00 grep slirp4netns

also slirp4netns is running.

I tried rebooting, but it didn't work.

I'll try stopping all containers and deleting $XDG_RUNTIME_DIR/netns/rootless-netns-*

Luap99 commented 10 months ago

If reboot didn't work you do not need to try and stop the containers. Slirp4netns is running and the tap0 interface is there so that looks good so the problem must be somewhere else. Do you have a ipv4 internet connection? I know ipv6 still causes some troubles. Also you can just run podman unshare --rootless-netns this should drop you in your shell in the this namespace so you can test if ping/curl works there correctly.

shenwii commented 10 months ago

@Luap99 Thanks a lot, that reminds me, the reason is that my routing is misconfigured.

containers / podman