Cannot start podman systemd units as core user: Error: error opening cidfile /run/<service>.service-cid

[nccurry]

[root@okd-utility ~]# rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 31.20200310.3.0 (2020-03-25T19:42:10Z)
                    Commit: 436592e6eb93e899bebab8dbd17514c85be683390ef8bbce8c6d96069ce4c543
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

  ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 31.20200223.3.0 (2020-03-12T01:20:02Z)
                    Commit: 3477082298eb5e2213433415c78f0919a991b40e62726b8fde434d244c1ec1b6
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

I cannot start podman systemd units on Fedora CoreOS as a nonroot user.

Some examples of two systemd units:

[Unit]
Description=CoreDNS
After=network-online.target
Wants=network-online.target

[Service]
User=core
Group=core
Restart=on-failure
ExecStartPre=/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
ExecStart=/usr/bin/podman run --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid -d -p 53:53 -v /home/core/coredns/:/root/:Z coredns/coredns:latest -conf /root/Corefile
ExecStop=/usr/bin/podman rm -f --cidfile /%t/%n-cid
KillMode=none
Type=forking
PIDFile=/%t/%n-pid

[Install]
WantedBy=multi-user.target

[Unit]
Description=HAProxy
After=network-online.target
Wants=network-online.target

[Service]
User=core
Group=core
Restart=on-failure
ExecStartPre=/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
ExecStart=/usr/bin/podman run --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid -d -p 80:80 -p 443:443 -p 6443:6443 -p 22623:22623 -v /home/core/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:Z haproxy:latest
ExecStop=/usr/bin/podman rm -f --cidfile /%t/%n-cid
KillMode=none
Type=forking
PIDFile=/%t/%n-pid

[Install]
WantedBy=multi-user.target

They both throw the following error when trying to start them:

Mar 27 16:44:49 okd-utility systemd[1]: coredns.service: Found left-over process 977 (podman pause) in control group while starting unit. Ignoring.
Mar 27 16:44:49 okd-utility systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Mar 27 16:44:49 okd-utility systemd[1]: Starting CoreDNS...
-- Subject: A start job for unit coredns.service has begun execution
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- A start job for unit coredns.service has begun execution.
-- 
-- The job identifier is 3570.
Mar 27 16:44:49 okd-utility systemd[1]: coredns.service: Found left-over process 977 (podman pause) in control group while starting unit. Ignoring.
Mar 27 16:44:49 okd-utility systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Mar 27 16:44:49 okd-utility podman[5462]: Error: error opening cidfile //run/coredns.service-cid
Mar 27 16:44:49 okd-utility systemd[1]: coredns.service: Control process exited, code=exited, status=125/n/a
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- An ExecStart= process belonging to unit coredns.service has exited.
-- 
-- The process' exit code is 'exited' and its exit status is 125.
Mar 27 16:44:49 okd-utility systemd[1]: coredns.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- The unit coredns.service has entered the 'failed' state with result 'exit-code'.
Mar 27 16:44:49 okd-utility systemd[1]: Failed to start CoreDNS.
-- Subject: A start job for unit coredns.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- A start job for unit coredns.service has finished with a failure.
-- 
-- The job identifier is 3570 and the job result is failed.

Mar 27 16:31:07 okd-utility systemd[1]: Starting HAProxy...
-- Subject: A start job for unit haproxy.service has begun execution
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- A start job for unit haproxy.service has begun execution.
-- 
-- The job identifier is 1349.
Mar 27 16:31:07 okd-utility podman[1421]: time="2020-03-27T16:31:07Z" level=warning msg="The cgroups manager is set to systemd but there is no systemd use>
Mar 27 16:31:07 okd-utility podman[1421]: time="2020-03-27T16:31:07Z" level=warning msg="For using systemd, you may need to login using an user session"
Mar 27 16:31:07 okd-utility podman[1421]: time="2020-03-27T16:31:07Z" level=warning msg="Alternatively, you can enable lingering with: `loginctl enable-li>
Mar 27 16:31:07 okd-utility podman[1421]: time="2020-03-27T16:31:07Z" level=warning msg="Falling back to --cgroup-manager=cgroupfs"
Mar 27 16:31:07 okd-utility podman[1421]: time="2020-03-27T16:31:07Z" level=warning msg="The cgroups manager is set to systemd but there is no systemd use>
Mar 27 16:31:07 okd-utility podman[1421]: time="2020-03-27T16:31:07Z" level=warning msg="For using systemd, you may need to login using an user session"
Mar 27 16:31:07 okd-utility podman[1421]: time="2020-03-27T16:31:07Z" level=warning msg="Alternatively, you can enable lingering with: `loginctl enable-li>
Mar 27 16:31:07 okd-utility podman[1421]: time="2020-03-27T16:31:07Z" level=warning msg="Falling back to --cgroup-manager=cgroupfs"
Mar 27 16:31:07 okd-utility podman[1421]: Error: error opening cidfile //run/haproxy.service-cid

The import bit seeming to be Error: error opening cidfile //run/haproxy.service-cid

If I remove the User= and Group= lines from the systemd unit files they will start just fine.

[dustymabe]

I'm not sure what the actual problem is here without looking a bit deeper but I will note that if you want to run rootless containers I think you are going to have to choose different ports (high ports) that aren't reserved for root to bind to on the host.. For example:

$ podman run -it -p 80:80 registry.fedoraproject.org/fedora:31
Error: failed to expose ports via rootlessport: "listen tcp 0.0.0.0:80: bind: permission denied\n"

[nccurry]

I am modifying the value in the file /proc/sys/net/ipv4/ip_unprivileged_port_start to the lowest port I need (53 in my case) in my ignition file to allow rootless containers to use privileged ports.

  "storage": {
    "files": [
      {
        "path": "/proc/sys/net/ipv4/ip_unprivileged_port_start",
        "contents": {
          "source": "data:,53"
        },
        "mode": 420,
        "user": {
          "name": "root"
        },
        "group": {
          "name": "root"
        }
      }
    ]
  },

[dustymabe]

I am modifying the value in the file /proc/sys/net/ipv4/ip_unprivileged_port_start to the lowest port I need (53 in my case) in my ignition file to allow rootless containers to use privileged ports.

  "storage": {
    "files": [
      {
        "path": "/proc/sys/net/ipv4/ip_unprivileged_port_start",
        "contents": {
          "source": "data:,53"
        },
        "mode": 420,
        "user": {
          "name": "root"
        },
        "group": {
          "name": "root"
        }
      }
    ]
  },

oh nice! Are you using a Live PXE or ISO for this? If not and you need this setting to be persistent across reboots you might want to put a file in /etc/sysctl.d/ to make it persistent. Something like this should do it:

$ sudo cat /etc/sysctl.d/11-lowports.conf
net.ipv4.ip_unprivileged_port_start=53

[lucab]

I think this is a symptom of https://github.com/containers/podman/issues/5572, whose larger umbrella of "better integration between podman and systemd" is being worked on upstream.

[dustymabe]

@nccurry - is this still reproducible?

[nccurry]

@dustymabe It is.

[root@fcos ~]# rpm-ostree status
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Sat 2022-04-02 23:58:34 UTC)
Deployments:
● fedora:fedora/x86_64/coreos/stable
                   Version: 35.20220313.3.1 (2022-03-28T19:24:40Z)
                    Commit: 4a21b7876e42f223bb70b00eedef698c34a8e72b5ffbfc597aec36f40d149a58
              GPGSignature: Valid signature by 787EA6AE1147EEE56C40B30CDB4639719867C58F

Steps to reproduce

# (as nonroot) Start running httpd container
podman run --name httpd -p 80:80 httpd:2.4

# (as nonroot) Generate systemd unit based on that running container
podman generate systemd --new httpd > /tmp/httpd.service

# (as nonroot) Stop template container
podman stop httpd
podman rm httpd

# (as root) Move systemd unit to /etc/
cat /tmp/httpd.service > /etc/systemd/system/httpd.service

# (as root) Edit systemd unit file to run as nonroot user by adding User= to [Service] stanza
vi /etc/systemd/system/httpd.service

# (as root) Start systemd unit
systemctl daemon-reload
systemctl start httpd.service

The contents of the systemd unit

# container-231ec973a2e792a7f9e78ed1c368b2a0c40ec4236e63c1fe0619957dc64c6625.service
# autogenerated by Podman 3.4.4
# Sun Apr  3 00:24:09 UTC 2022

[Unit]
Description=Podman container-231ec973a2e792a7f9e78ed1c368b2a0c40ec4236e63c1fe0619957dc64c6625.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers

[Service]
User=user
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm -f %t/%n.ctr-id
ExecStart=/usr/bin/podman run --cidfile=%t/%n.ctr-id --cgroups=no-conmon --rm --sdnotify=conmon -d --replace --name httpd -p 80:80 httpd:2.4
ExecStop=/usr/bin/podman stop --ignore --cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm -f --ignore --cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all

[Install]
WantedBy=default.target

Error message:

Apr 03 00:29:05 fcos podman[197443]: Error: error opening cidfile /run/httpd.service.ctr-id
Apr 03 00:29:05 fcos systemd[1]: httpd.service: Main process exited, code=exited, status=125/n/a