hello-app example not started on Debian Bookworm

[ancebfer]

Describe the bug hello-app example (from examples/hello-app/ folder) not started on Debian Bookworm. It always finish with error: Error: unable to start container X: netavark: No such file or directory (os error 2)

It doesn't seem to be a bug in podman because the example runs correctly when launched directly with podman:

podman  run --name hello-app -p 8080:8080 gcr.io/google-samples/hello-app:1.0 
2023/01/01 10:34:44 Server listening on port 8080
2023/01/01 10:35:12 Serving request: /
2023/01/01 10:35:12 Serving request: /favicon.ico

To Reproduce Steps to reproduce the behavior:

• Running podman-compose in the example folder (https://github.com/containers/podman-compose/tree/devel/examples/hello-app):

~/podman-compose-devel/examples/hello-app$ podman-compose up
['podman', '--version', '']
using podman version: 4.3.1
** excluding:  set()
['podman', 'network', 'exists', 'hello-app_default']
podman create --name=hello-app_web_1 --label io.podman.compose.config-hash=123 --label io.podman.compose.project=hello-app --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=hello-app --label com.docker.compose.project.working_dir=/home/devel/podman-compose-devel/examples/hello-app --label com.docker.compose.project.config_files=docker-compose.yaml --label com.docker.compose.container-number=1 --label com.docker.compose.service=web --net hello-app_default --network-alias web -p 8080:8080 gcr.io/google-samples/hello-app:1.0
945f51fa502fc70dfc60ea2b05a2497d6fc0b09506641eadd68cd7e1d6b740a7
exit code: 0
podman start -a hello-app_web_1
Error: unable to start container 945f51fa502fc70dfc60ea2b05a2497d6fc0b09506641eadd68cd7e1d6b740a7: netavark: No such file or directory (os error 2)
exit code: 125

• Running master podman-compose in the example folder (https://github.com/containers/podman-compose/tree/devel/examples/hello-app):

~/podman-compose-devel/examples/hello-app$ ../../podman_compose.py up
podman-compose version: 1.0.4
['podman', '--version', '']
using podman version: 4.3.1
** excluding:  set()
['podman', 'ps', '--filter', 'label=io.podman.compose.project=hello-app', '-a', '--format', '{{ index .Labels "io.podman.compose.config-hash"}}']
podman pod create --name=pod_hello-app --infra=false --share=
1a94b6303e85e16732c6d514cc61a91228cca7a28151d026dd27f10a7ebb0029
exit code: 0
['podman', 'network', 'exists', 'hello-app_default']
podman create --name=hello-app_web_1 --pod=pod_hello-app --label io.podman.compose.config-hash=ef191a8214ebad4a7d3c0c6981f2437bde31ed9a951a6db0b44a6aabe1e76d3d --label io.podman.compose.project=hello-app --label io.podman.compose.version=1.0.4 --label PODMAN_SYSTEMD_UNIT=podman-compose@hello-app.service --label com.docker.compose.project=hello-app --label com.docker.compose.project.working_dir=/home/devel/podman-compose-devel/examples/hello-app --label com.docker.compose.project.config_files=docker-compose.yaml --label com.docker.compose.container-number=1 --label com.docker.compose.service=web --net hello-app_default --network-alias web -p 8080:8080 gcr.io/google-samples/hello-app:1.0
b012d239f6916508f0aec7a0b02f3b3bd198354f950ea26ee17c83c2571523ef
exit code: 0
podman start -a hello-app_web_1
Error: unable to start container b012d239f6916508f0aec7a0b02f3b3bd198354f950ea26ee17c83c2571523ef: netavark: No such file or directory (os error 2)
exit code: 125

• Running master podman-compose in the example folder (https://github.com/containers/podman-compose/tree/devel/examples/hello-app) disabling pod creation:

~/podman-compose-devel/examples/hello-app$ ../../podman_compose.py --no-pod up
podman-compose version: 1.0.4
['podman', '--version', '']
using podman version: 4.3.1
** excluding:  set()
['podman', 'ps', '--filter', 'label=io.podman.compose.project=hello-app', '-a', '--format', '{{ index .Labels "io.podman.compose.config-hash"}}']
['podman', 'network', 'exists', 'hello-app_default']
podman create --name=hello-app_web_1 --label io.podman.compose.config-hash=ef191a8214ebad4a7d3c0c6981f2437bde31ed9a951a6db0b44a6aabe1e76d3d --label io.podman.compose.project=hello-app --label io.podman.compose.version=1.0.4 --label PODMAN_SYSTEMD_UNIT=podman-compose@hello-app.service --label com.docker.compose.project=hello-app --label com.docker.compose.project.working_dir=/home/devel/podman-compose-devel/examples/hello-app --label com.docker.compose.project.config_files=docker-compose.yaml --label com.docker.compose.container-number=1 --label com.docker.compose.service=web --net hello-app_default --network-alias web -p 8080:8080 gcr.io/google-samples/hello-app:1.0
2f33dc3ada530cb6f4aaa9e3502df66334cd22799ceb58089ab9d029905b8b2f
exit code: 0
podman start -a hello-app_web_1
Error: unable to start container 2f33dc3ada530cb6f4aaa9e3502df66334cd22799ceb58089ab9d029905b8b2f: netavark: No such file or directory (os error 2)
exit code: 125

Expected behavior "podman-compose up" should start the container without errors.

Actual behavior "podman-compose up" doesn't start the container with errors: Error: unable to start container X: netavark: No such file or directory (os error 2)

Output

~/podman-compose-devel/examples/hello-app$ podman-compose version
['podman', '--version', '']
using podman version: 4.3.1
podman-composer version  1.0.3
podman --version 
podman version 4.3.1
exit code: 0

~/podman-compose-devel/examples/hello-app$ ../../podman_compose.py version
podman-compose version: 1.0.4
['podman', '--version', '']
using podman version: 4.3.1
podman-compose version 1.0.4
podman --version 
podman version 4.3.1
exit code: 0

~/podman-compose-devel/examples/hello-app$ ../../podman_compose.py --no-pod up
podman-compose version: 1.0.4
['podman', '--version', '']
using podman version: 4.3.1
** excluding:  set()
['podman', 'ps', '--filter', 'label=io.podman.compose.project=hello-app', '-a', '--format', '{{ index .Labels "io.podman.compose.config-hash"}}']
['podman', 'network', 'exists', 'hello-app_default']
podman create --name=hello-app_web_1 --label io.podman.compose.config-hash=ef191a8214ebad4a7d3c0c6981f2437bde31ed9a951a6db0b44a6aabe1e76d3d --label io.podman.compose.project=hello-app --label io.podman.compose.version=1.0.4 --label PODMAN_SYSTEMD_UNIT=podman-compose@hello-app.service --label com.docker.compose.project=hello-app --label com.docker.compose.project.working_dir=/home/devel/podman-compose-devel/examples/hello-app --label com.docker.compose.project.config_files=docker-compose.yaml --label com.docker.compose.container-number=1 --label com.docker.compose.service=web --net hello-app_default --network-alias web -p 8080:8080 gcr.io/google-samples/hello-app:1.0
c0307791f08d45c2863c80e841d6a965b744de5adef02322acf75c8fe55a03bb
exit code: 0
podman start -a hello-app_web_1
Error: unable to start container c0307791f08d45c2863c80e841d6a965b744de5adef02322acf75c8fe55a03bb: netavark: No such file or directory (os error 2)
exit code: 125

~/podman-compose-devel/examples/hello-app$ podman network inspect hello-app_default 
[
     {
          "name": "hello-app_default",
          "id": "e221f9905f77601496e2ed2944a90834bd59ae7b23845ee05d594013e0791fb8",
          "driver": "bridge",
          "network_interface": "podman2",
          "created": "2023-01-01T11:25:03.323115063+01:00",
          "subnets": [
               {
                    "subnet": "10.89.1.0/24",
                    "gateway": "10.89.1.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "labels": {
               "com.docker.compose.project": "hello-app",
               "io.podman.compose.project": "hello-app"
          },
          "ipam_options": {
               "driver": "host-local"
          }
     }
]

$ podman version
Client:       Podman Engine
Version:      4.3.1
API Version:  4.3.1
Go Version:   go1.19.4
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

$ podman info
host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.3+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.3, commit: unknown'
  cpuUtilization:
    idlePercent: 94.23
    systemPercent: 2.99
    userPercent: 2.78
  cpus: 8
  distribution:
    codename: bookworm
    distribution: debian
    version: unknown
  eventLogger: journald
  hostname: debian
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.0.0-6-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 170352640
  memTotal: 8340271104
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.5+dfsg-1+b1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 1073201152
  swapTotal: 1073737728
  uptime: 2h 7m 4.00s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/devel/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/devel/.local/share/containers/storage
  graphRootAllocated: 67316002816
  graphRootUsed: 17080094720
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 30
  runRoot: /run/user/1000/containers
  volumePath: /home/devel/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

Additional context

https://github.com/containers/podman/issues/16956 https://github.com/containers/podman/issues/16958

[ancebfer]

Found problem with strace:

execve("/usr/local/bin/iptables", ["iptables", "--version"], 0x7fff18f1eb88 /* 46 vars */) = -1 ENOENT (No existe el fichero o el directorio)
execve("/usr/bin/iptables", ["iptables", "--version"], 0x7fff18f1eb88 /* 46 vars */) = -1 ENOENT (No existe el fichero o el directorio)
execve("/bin/iptables", ["iptables", "--version"], 0x7fff18f1eb88 /* 46 vars */) = -1 ENOENT (No existe el fichero o el directorio)
execve("/usr/local/games/iptables", ["iptables", "--version"], 0x7fff18f1eb88 /* 46 vars */) = -1 ENOENT (No existe el fichero o el directorio)
execve("/usr/games/iptables", ["iptables", "--version"], 0x7fff18f1eb88 /* 46 vars */) = -1 ENOENT (No existe el fichero o el directorio)
execve("/usr/sbin/iptables", ["iptables", "--version"], 0x7fff18f1eb88 /* 46 vars */) = -1 ENOENT (No existe el fichero o el directorio)
exit_group(127)                         = ?
+++ exited with 127 +++

Debian podman package (https://packages.debian.org/bookworm/podman) only suggest iptables package therefore it is not installed by default.

After installing iptables podman-compose works right:

~/podman-compose-devel/examples/hello-app$ podman-compose up -d
['podman', '--version', '']
using podman version: 4.3.1
** excluding:  set()
['podman', 'network', 'exists', 'hello-app_default']
podman run --name=hello-app_web_1 -d --label io.podman.compose.config-hash=123 --label io.podman.compose.project=hello-app --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=hello-app --label com.docker.compose.project.working_dir=/home/devel/podman-compose-devel/examples/hello-app --label com.docker.compose.project.config_files=docker-compose.yaml --label com.docker.compose.container-number=1 --label com.docker.compose.service=web --net hello-app_default --network-alias web -p 8080:8080 gcr.io/google-samples/hello-app:1.0
4aa5362e8ee96708284d4850e093b9fd7833652316985a2882d187fe454258fd
exit code: 0

~/podman-compose-devel/examples/hello-app$ podman-compose logs
['podman', '--version', '']
using podman version: 4.3.1
podman logs hello-app_web_1
2023/01/01 18:03:34 Server listening on port 8080
exit code: 0

[brianm]

For the next person who runs into this, iptables must not only be installed, but be on $PATH when running it as root -- ensure /sbin is on $PATH when running iptables. When running as a regular user, it doesn't seem to matter. How bizarre.