Startup of podman-restart conflicts with podman-kube template if both are used

Jyckle commented 3 months ago

Issue Description

I ran into an odd issue, where on a system using both the podman-restart.service and multiple podman-kube@.service files, the normally running container (just using --restart=always) would not run on reboot. After diving in, the errors indicate that this is happening because podman-kube does a replace, and podman-restart was trying to start containers that had already been replaced by podman-kube

Steps to reproduce the issue

Set up a normal container with --restart=always
Set up a pod using the podman-kube template
sudo systemctl enable podman-restart
Reboot the machine
View output of podman container ps -a and observe non-pod containers not starting

Note that adding multiple different normal containers and multiple different podman-kube templates will likely make this more reproducible, as it is intermittent and based on the timing as far as I can tell.

Describe the results you received

The podman-kube pods start up fine, but the normal container with --restart=always does not restart systemctl status podman-restart show that the command failed on a container id that does not exist, but was a member of a pod before the restart

Describe the results you expected

All containers restart successfully

podman info output

Sadly, I can't provide the output of podman info, but here's the details I can provide
podman version: 4.4.1
os: Rhel 8

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

Additional information

In my testing, by filtering out any containers that have the label 'PODMAN_SYSTEMD_UNIT' I was able to resolve this issue. I think there might be a more elegant solution overall, especially now that there seems to be label! filtering capability. I am also not a systemd expert, so there might be a more elegant solution to simply order the two systemd unit files to not overlap. That being said, I think changing this line: https://github.com/containers/podman/blob/main/contrib/systemd/system/podman-restart.service.in#L12 To look like: ExecStart=@@PODMAN@@ $LOGGING start --all --filter restart-policy=always --filter label!=PODMAN_SYSTEMD_UNIT might be sufficient to solve the problem!

rhatdan commented 3 months ago

@ygalblum @vrothberg WDYT?

Should kube yaml containers have restart=always set?

rhatdan commented 3 months ago

Also is there a way to make sure podman-restart is run before any of the quadlets start?

vrothberg commented 3 months ago

We should probably exclude "kube" containers from the podman-restart.service.

ygalblum commented 3 months ago

Should kube yaml containers have restart=always set?

@rhatdan This has always been on open question because there are basically two restart handlers (podman and systemd). But, we can't prevent users from doing it (keep in mind that Quadlet does not read the YAML file and moreover the user can change the if without Quadlet knowing about it). So, we just need to deal with it.

We should probably exclude "kube" containers from the podman-restart.service.

@vrothberg When you say "kube" containers, do you mean ones that originated from a Quadlet .kube file or any container started by kube play? I don't think there is a simple way to figure out the former.

vrothberg commented 3 months ago

We can focus on the ones from a .kube file.

github-actions[bot] commented 2 months ago

A friendly reminder that this issue had no activity for 30 days.

containers / podman