Closed bfallik closed 4 years ago
After reading the escape-from-the-dock blog post I was inspired to try dropping the --detach
podman flag and changing the unit file to type=simple
and, voila, logging is back.
I can understand why type=forking
may be incorrect for coredns since it probably doesn't fork but I'm confused how the top
example from the blog post behaves differently. According to strace
top
doesn't fork
, at least as tested on my Fedora host.
I don't actually know how type=forking
works with a pidfile specified - but I know that the reason we moved away was because tracking podman run
is not correct behavior, and can lead to VERY bad results.
Our containers are not direct children of Podman. During container startup, we launch an instance of Conmon (an extremely lightweight monitor process) which double-forks to daemonize and then launches the container as its own child. As a result of this, once the container is running, you can hit podman run
with SIGKILL, and it will go away - but the container will still be running fine, in the background. Systemd doesn't know that, and also can't see it with cgroups (the container makes its own cgroups), so it interpreted the podman run
process dying as the container dying, when it was still running - the states of Podman and systemd desynced, and nothing worked properly afterwards without manual intervention. The use of a pidfile might be able resolve this, but I'm not sure it will - we've never tested it that way.
Interesting... thanks for sharing that.
A friendly reminder that this issue had no activity for 30 days.
I think @mheon answered this one, Closing. Reopen if I am mistaken.
What is the current best practice for getting container output in the journal when running rootless containers as user units? With podman 3.0.1 on Debian 11, I see
podman generate systemd --new ...
user unit_SYSTEMD_USER_UNIT={unit_name}
when -d
is removed and type=simple
is setIn the latter case, I tried killing the podman run
process and everything appeared to shut down cleanly (exit 0/SUCCESS
in systemd and no pod sticking around either). The only problem was that the unit didn't restart because it's set to on-failure
, but that's easily fixable.
@vrothberg PTAL
I do not recommend using type=simple
for the reasons @mheon has mentioned above.
What is the current best practice for getting container output in the journal when running rootless containers as user units?
I do see the stdout in the journal. For instance, a container that does nothing more than echo 123
:
[...] Jan 10 14:52:09 nebuchadnezzar conmon[163851]: 123 [...]
Are you not seeing that on your machine?
I was definitely not seeing output in the journal from configurations like those mentioned above. However, I have since changed my units to be like:
[Unit]
...
[Service]
...
ExecStart=/usr/bin/podman run --cidfile=... --cgroups=no-conmon --rm --pod-id-file ... --sdnotify=conmon -d --replace --log-driver=journald --name... ...
...
Type=notify
NotifyAccess=all
...
Things have been working as I would like. In particular, I think the --log-driver=journald
was the solution here. Since I've kept -d
and Type=notify
I think this doesn't fall afoul of any of the problems mentioned above. Does this make sense or is there something incorrect with this configuration as well?
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
I was experimenting converting an existing systemd+podman container to the approach described in Running containers with Podman and shareable systemd services.
I can start and stop the container and confirm the service is behaving as normal except for one gap. Logs written to STDOUT within the container are not being propagated to journald on the host.
Steps to reproduce the issue:
systemctl start xxx.service
journalctl -u xxx.service -n 100
Describe the results you received:
Describe the results you expected:
I can see the logs in podman:
Additional information you deem important (e.g. issue happens only occasionally):
I don't know if this is related by the article claims that podman starts several processes within the cgroup, but
systemctl status
only shows the conmon main process.Below is my unit file
Output of
podman version
:Output of
podman info --debug
:Package info (e.g. output of
rpm -q podman
orapt list podman
):Additional environment details (AWS, VirtualBox, physical, etc.):
running on physical hardware