[Bug]: Permission issues publishing ports via quadlet

[jwillikers]

Issue Description

Running a podman container using quadlet hits permission issues in the container when attempting to bind to port 80 / 443. This same container has no issues when run directly with podman-run.

Steps to reproduce the issue

Steps to reproduce the issue

1. Create the quadlet network file ~/.config/containers/systemd/podman1.network.

[Network]
IPv6=yes

3. Create the quadlet container file ~/.config/containers/systemd/caddy.container.

[Unit]
Description=Caddy reverse proxy
Before=local-fs.target

[Container]
ContainerName=caddy
Image=quay.io/jwillikers/caddy-cloudflare:latest
Label=io.containers.autoupdate=registry
Network=podman1.network
PublishPort=80:80
PublishPort=443:443
RemapUsers=keep-id
PodmanArgs=--secret=CLOUDFLARE_API_TOKEN,type=env
Volume=/home/jordan/Projects/quartz64-caddy-config/Caddyfile:/etc/caddy/Caddyfile:Z
Volume=caddy-config:/config:Z
Volume=caddy-data:/data:Z

[Service]
Restart=always

[Install]
WantedBy=multi-user.target default.target

4. Start the service.

systemctl --user start caddy.service

Describe the results you received

The container was constantly restarted because it failed to bind to port 80 / 443.

Describe the results you expected

I expected the container to just work as it does when running with podman run:

podman run \
                                   --detach \
                                   --label "io.containers.autoupdate=registry" \
                                   --name caddy \
                                   --network systemd-podman1 \
                                   --publish 80:80/tcp \
                                   --publish 443:443/tcp \
                                   --rm \
                                   --secret CLOUDFLARE_API_TOKEN,type=env \
                                   --volume $PWD/Caddyfile:/etc/caddy/Caddyfile:Z \
                                   --volume caddy-config:/config/:Z \
                                   --volume caddy-data:/data/:Z \
                                   quay.io/jwillikers/caddy-cloudflare:latest

podman info output

host:
  arch: arm64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.5-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: c9f7f19eb82d5b8151fc3ba7fbbccf03fdcd0325'
  cpuUtilization:
    idlePercent: 78.41
    systemPercent: 9.37
    userPercent: 12.22
  cpus: 4
  distribution:
    distribution: '"manjaro-arm"'
    version: unknown
  eventLogger: journald
  hostname: quartz64.jwillikers.io
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.1.9-1-MANJARO-ARM
  linkmode: dynamic
  logDriver: journald
  memFree: 6697742336
  memTotal: 8053100544
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.8-1
    path: /usr/bin/crun
    version: |-
      crun version 1.8
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 12079648768
  swapTotal: 12079648768
  uptime: 0h 46m 41.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/jordan/.config/containers/storage.conf
  containerStore:
    number: 8
    paused: 0
    running: 4
    stopped: 4
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/jordan/.local/share/containers/storage
  graphRootAllocated: 30523527168
  graphRootUsed: 10186518528
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 53
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/jordan/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.0
  Built: 1675482507
  BuiltTime: Fri Feb  3 21:48:27 2023
  GitCommit: 3443f453e28169a88848f90a7ce3137fc4a4bebf-dirty
  GoVersion: go1.20
  Os: linux
  OsArch: linux/arm64
  Version: 4.4.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

aarch64, Quartz64, Manjaro

Additional information

The same issue occurs using a generated kubernetes YAML with a caddy.kube quadlet file.

[rhatdan]

Run with --dryrun to see what the output is.

[rhatdan]

Looks like you are attempting to run a container that will bind to port 80 as a rootless user. This will never be allowed on the host system

[jwillikers]

So, I can bind to port 80 and 443 with my rootless Caddy container when running the container via podman run and also as a systemd service generated by podman generate systemd. I've set net.ipv4.ip_unprivileged_port_start to 80 to make this possible.

$ sudo sysctl -n net.ipv4.ip_unprivileged_port_start
80

However, I don't understand why it is failing when running via quadlet with the same configuration. Even if I change the host ports to be higher, i.e. PublishPort=8080:80 and PublishPort=8443:443, it still seems to fail inside the container when binding to the container's port 80 / `443. The following error message comes from inside the container.

http app module: start: listening on :443: listen tcp :443: bind: permission denied

[ygalblum]

It seems that you are failing inside the container and not on the host network. I did not try it yet, but could this RemapUsers=keep-id be the reason? I didn't see that you are passing the equivalent flag when using podman run

[jwillikers]

@ygalblum Right! I just figured that out. I assumed that RemapUsers=keep-id was the equivalent of passing --userns keep-id to podman run on the command-line.

[ygalblum]

@jwillikers It is, but you're not setting that in your podman run command. If I'm not mistaken the issue here is that you keep the user id from the host, but do not keep the configuration of net.ipv4.ip_unprivileged_port_start. As a result, you have a non-privileged user trying to bind to a privileged port inside the container

[jwillikers]

You're right, my mistake there. For some season I thought I was using that in my Podman run command since I use it everywhere else. Thanks!