if --network host and -p ports are used together later "podman generate systemd" generates failing service

rugk commented 3 years ago

copied from and originally reported at https://github.com/containers/podman/issues/10882

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When you use host networking and ports together, the systemd units podman generates are broken in such a way that if the ports that are passed via -p are actually used/bount inside of the container, the container/service will fail, because they are already in use by an infra container – that previously was not there/block it.

Steps to reproduce the issue:

The issue happens when you use network_mode = "host" and ports at the same time (I know, ports are ignored then) and let podman generate a systemd unit based on that.

For the example to work, you need a simple caddy server that binds on ports 80 and 443… So create a Caddyfile e.g.

So e.g. create a YML docker-compose.yml file…

version: "3.7"

services:

  caddy:
    image: caddy
    restart: unless-stopped
    network_mode: "host"
    # ports are ignored, because of host networking
    ports:
      - "80:80"
      - "443:443"
      # […]
    volumes:
      - caddy_data:/data
      - caddy_config:/config
      # […]
    environment:
      - HOST_DOMAIN=127.0.0.1
      - MAIL_ACME=${MAIL_ACME:?MAIL_ACME variable missing}
    labels:
      - io.containers.autoupdate=registry

volumes:
  caddy_data:
    # always persist volume by forcing external creation
    # https://docs.docker.com/compose/compose-file/compose-file-v3/#external
    external: true
  caddy_config:

Run podman-compose:

$ podman-compose -p caddy --dry-run up
podman pod create --name=caddy --share net -p 80:80 -p ***:*** -p 443:443
podman volume inspect caddy_caddy_data || podman volume create caddy_caddy_data
podman volume inspect caddy_caddy_config || podman volume create caddy_caddy_config
podman create --name=caddy_caddy_1 --pod=caddy --label io.containers.autoupdate=registry --label io.podman.compose.config-hash=123 --label io.podman.compose.project=caddy --label io.podman.compose.version=0.0.1 --label com.docker.compose.container-number=1 --label com.docker.compose.service=caddy --network host -e HOST_DOMAIN=127.0.0.1 -e MAIL_ACME=***** -v caddy_caddy_data:/data -v caddy_caddy_config:/config [***some volume mounts removed***] --add-host caddy:127.0.0.1 --add-host caddy_caddy_1:127.0.0.1 --restart unless-stopped caddy

Or obviously just run the podman command shown here.

podman generate systemd "caddy" --restart-policy=always --new --name --files
Move files to $HOME/.config/systemd/user and enable/start the services.

Describe the results you received:

# pod-caddy.service
# autogenerated by Podman 3.2.0
# Thu Jul  8 21:34:38 CEST 2021

[Unit]
Description=Podman pod-caddy.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target
RequiresMountsFor=
Requires=container-caddy_caddy_1.service
Before=container-caddy_caddy_1.service

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=always
TimeoutStopSec=70
ExecStartPre=/bin/rm -f %t/pod-caddy.pid %t/pod-caddy.pod-id
ExecStartPre=/usr/bin/podman pod create --infra-conmon-pidfile %t/pod-caddy.pid --pod-id-file %t/pod-caddy.pod-id --name=caddy --share net -p 80:80 -p 443:443 -p 1234:1234 --replace
ExecStart=/usr/bin/podman pod start --pod-id-file %t/pod-caddy.pod-id
ExecStop=/usr/bin/podman pod stop --ignore --pod-id-file %t/pod-caddy.pod-id -t 10
ExecStopPost=/usr/bin/podman pod rm --ignore -f --pod-id-file %t/pod-caddy.pod-id
PIDFile=%t/pod-caddy.pid
Type=forking

[Install]
WantedBy=multi-user.target default.target

# /var/home/c-caddy/.config/systemd/user/container-caddy_caddy_1.service
# container-caddy_caddy_1.service
# autogenerated by Podman 3.2.0
# Thu Jul  8 21:34:38 CEST 2021

[Unit]
Description=Podman container-caddy_caddy_1.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=always
TimeoutStopSec=70
ExecStartPre=/bin/rm -f %t/container-caddy_caddy_1.pid %t/container-caddy_caddy_1.ctr-id
ExecStart=/usr/bin/podman run --conmon-pidfile %t/container-caddy_caddy_1.pid --cidfile %t/container-caddy_caddy_1.ctr-id --cgroups=no-conmon --pod-id-file %t/pod-caddy.pod-id -d --replace --name=caddy_caddy_1 --label io.containers.autoupdate=registry --label io.podman.compose.config-hash=123 --label io.podman.compose.project=caddy --label io.podman.compose.version=0.0.1 --label com.docker.compose.container-number=1 --label com.docker.compose.service=caddy --network host -e HOST_DOMAIN=127.0.0.1 -e MAIL_ACME=********** -v caddy_caddy_data:/data -v caddy_caddy_config:/config [***some volume mounts removed***] --add-host caddy:127.0.0.1 --add-host caddy_caddy_1:127.0.0.1 --restart unless-stopped caddy
ExecStop=/usr/bin/podman stop --ignore --cidfile %t/container-caddy_caddy_1.ctr-id -t 10
ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile %t/container-caddy_caddy_1.ctr-id
PIDFile=%t/container-caddy_caddy_1.pid
Type=forking

[Install]
WantedBy=multi-user.target default.target

What happens:

Now what happens when you systemctl --user start pod-caddy.service is that the caddy service will fail, because it cannot bind to the given ports. The reason is just that the ifra/pod container there already uses the ports it wants to bind.

Describe the results you expected:

It should not fail. It did not fail when testing it with podman-compose, so now it also should not fail.

Additional information you deem important (e.g. issue happens only occasionally):

AFAIK this is not an issue in podman-compose, because the spec for docker compose says:

Port mapping is incompatible with network_mode: host

So AFAIK, Docker also shows no error if you use these two things together. They should just be silently ignored.

Also if you still argue this is a user error, I'd say that even if it is, it should then consistently fail. I.e. either should the podman-compose/podman start fail initially or nothing should fail. But failing only on the generated systemd services is very bad IMHO and it took me quite some time to troubleshoot/find out what the actual issue here is.

Output of podman version:

$ podman-compose version
using podman version: podman version 3.2.0
podman-composer version  0.1.7dev
podman --version
podman version 3.2.0
$ podman version
Version:      3.2.0
API Version:  3.2.0
Go Version:   go1.16.3
Built:        Wed Jun  9 16:24:16 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

```yml host: arch: amd64 buildahVersion: 1.21.0 cgroupControllers: [] cgroupManager: systemd cgroupVersion: v2 conmon: package: conmon-2.0.27-2.fc34.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.27, commit: ' cpus: 4 distribution: distribution: fedora version: "34" eventLogger: journald hostname: minipure idMappings: gidmap: - container_id: 0 host_id: 1003 size: 1 - container_id: 1 host_id: 296608 size: 65536 uidmap: - container_id: 0 host_id: 1003 size: 1 - container_id: 1 host_id: 296608 size: 65536 kernel: 5.12.9-300.fc34.x86_64 linkmode: dynamic memFree: 50017533952 memTotal: 67307839488 ociRuntime: name: crun package: crun-0.20.1-1.fc34.x86_64 path: /usr/bin/crun version: |- crun version 0.20.1 commit: 0d42f1109fd73548f44b01b3e84d04a279e99d2e spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL os: linux remoteSocket: path: /run/user/1003/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: true serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.1.9-1.fc34.x86_64 version: |- slirp4netns version 1.1.8+dev commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876 libslirp: 4.4.0 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.0 swapFree: 4294963200 swapTotal: 4294963200 uptime: 32h 18m 18.39s (Approximately 1.33 days) registries: search: - registry.fedoraproject.org - registry.access.redhat.com - docker.io - quay.io store: configFile: /var/home/*****/.config/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: overlay.mount_program: Executable: /usr/bin/fuse-overlayfs Package: fuse-overlayfs-1.5.0-1.fc34.x86_64 Version: |- fusermount3 version: 3.10.3 fuse-overlayfs: version 1.5 FUSE library version 3.10.3 using FUSE kernel interface version 7.31 graphRoot: /var/home/*****/.local/share/containers/storage graphStatus: Backing Filesystem: btrfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "false" imageStore: number: 2 runRoot: /run/user/****/containers volumePath: /var/home/*****/.local/share/containers/storage/volumes version: APIVersion: 3.2.0 Built: 1623248656 BuiltTime: Wed Jun 9 16:24:16 2021 GitCommit: "" GoVersion: go1.16.3 OsArch: linux/amd64 Version: 3.2.0 ```

Package info (e.g. output of rpm -q podman or apt list podman):

As there is no such thing on rpm-ostree

$ rpm -q podman
podman-3.2.0-5.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes, version is only a minor release/slightly older.

Additional environment details (AWS, VirtualBox, physical, etc.): physical

Fedora CoreOS 34 (stable stream)

$ rpm-ostree status
State: idle
warning: Failed to query journal: couldn't find current boot in journal
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Thu 2021-07-08 20:07:37 UTC)
Deployments:
● ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 34.20210611.3.0 (2021-06-28T15:43:17Z)
                BaseCommit: 0bdf0aee2585cafb224be19eec3b77501cb2044f028cf43a78f4de7ebd7c1a47
              GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39
           LayeredPackages: **** podman-compose

rugk commented 3 years ago

Host networking and port forwarding are incompatible and if the pod were to be created with --net=host it would not let you add ports to it; by adding net=host only to the container, now the container has not joined the pod's network namespace, and the container service and pod are competing for access to the ports (the pod, being started first, always wins). Podman is doing exactly as podman-compose instructed it to do, and if there is a bug it's in how compose YAML with net=host and port forwarding is handled.

-> The point here seems to be that net should not be added to the container, but to the pod, podman-compose adds it to the container only though…

rugk commented 3 years ago

Note I guess the reason this happens is actually the same as https://github.com/containers/podman-compose/issues/316. So feel free to close this as a duplicate of https://github.com/containers/podman-compose/issues/316.

muayyad-alsadi commented 2 years ago

I believe this is resolved in latest code, please confirm, if not reopen please.

rugk commented 2 years ago

and #316 then too?

containers / podman-compose

if --network host and -p ports are used together later "podman generate systemd" generates failing service #314