systemctl does not work inside the toolbox

[mildred]

I'm setting up a toolbox for a development environment for a web application that requires postgresql running. I'm installing postgresql-server on the toolbox and then trying to start it using the usual method:

INS mildred@toolbox$ sudo systemctl start postgresql.service
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down

This is understandable as I'm running within a container, and systemd usually is not started in containers. Still, it means I have to manually start up postgresql by reading the systemd service file.

It would be great if systemd was started as entrypoint in the toolbox image, instead of sleep +Inf. Of course, udev and low level services would have to be prevented from starting so that a blank toolbox would not start any service.

[mildred]

Trying out running systemd with the toolbox, I find a problem with it. systemd is not running as PID 1 and tries to start in user mode and not in system mode.

This is because there is the --pid=host option passed to podman create.

May I suggest that podman create keeps the PID namespace (so systemd can run) but podman exec gets a new option to keep the PID namespace of the host. it is possible with nsenter, why not podman exec ?

found a nice post here: https://developers.redhat.com/blog/2019/04/24/how-to-run-systemd-in-a-container/

[mildred]

Trying to use nsenter to have systemd running in a PID namespace and toolbox shells running in the host PID namespace does not work so well. the /proc filesystem mounted within the container represents the container PID namespace and not the host. As such, commands are complaining with errors like:

ls: cannot read symbolic link '/proc/self': No such file or directory

Perhaps we can have different features enabled for the toolboxes, and we can choose for each toolbox if we want to

• run an init system but have a separate PID namespace
• do not run an init system and keep the same PID namespace

It corresponds to two different use cases.

[mildred]

There are a few other issues running systemd in the container, systemd is mounting tmpfs to /run, overriding bind mounts that were there.

[mildred]

It also means we cannot share the system dbus socket inside the toolbox because the system bus needs to points to the toolbox system instead. I don't know what are the consequence of that.

[debarshiray]

There are a few other issues running systemd in the container, systemd is mounting tmpfs to /run, overriding bind mounts that were there.

I thought podman create --systemd ... is meant to take care of that. Maybe it doesn't work rootless because /run inside the container appears to be fuse-overlayfs?

It also means we cannot share the system dbus socket inside the toolbox because the system bus needs to points to the toolbox system instead. I don't know what are the consequence of that.

I don't know how useful having the system D-Bus socket inside the container is, in practice. I added it so that one could at least browse through the list of names using d-feet spawned from the container.

How insane would it be to run a second instance of the bus inside the container for systemd and then have a systemctl wrapper point to it? :)

[mildred]

There are a few other issues running systemd in the container, systemd is mounting tmpfs to /run, overriding bind mounts that were there.

I found a fix for that, just add --tmpfs /run before any other volume definition and it mounts a tmpfs there. systemd is clever enough to detect it and avoids mounting a new tmpfs over.

I don't know how useful having the system D-Bus socket inside the container is, in practice. I added it so that one could at least browse through the list of names using d-feet spawned from the container.

How insane would it be to run a second instance of the bus inside the container for systemd and then have a systemctl wrapper point to it? :)

What I did in the referenced pull request is that I let the user choose at toolbox creation time. Either you want to spawn an init system in the toolbox (--init flag) and you get the toolbox dBus socket inside the toolbox. or you don't and you have access to PIDs and system dBus from the host. It's difficult to imagine having both.

[yuriploc]

@mildred Is there a solution yet? I'm trying to migrate from Docker to Podman in my development environment and would like to work with Toolbox and Postgres.

[mildred]

Will soon have to do the same, hence this feature request. I have a pull request you can test here https://github.com/debarshiray/toolbox/pull/177 but I do not guarantee it works well.

Basically, this is a different mode of operation and many things do not work the same. Traditional toolbox allows you to connect to the host system (host dBus in particular) and if running systemd within the toolbox, this has to change. So the toolbox with systemd must be created differently.

I'm just thinking that the future might be with systemd portable services you run on the host, and access them on the toolbox (as the toolbox shares the same network namespace)

[mildred]

Well I tried other solutions to run postgresql, and the toolbox is the best solution I could find. The PR still works and I just updated it to make use of it on top of latest toolbox.

[ibotty]

I do think that better integration between the host's systemd and a systemd (user or system) inside toolboxes is worthwhile, but have you considered running PostgreSQL in a podman container instead? That will be similar to how some people run PostgreSQL in production on real servers so it is rather well tested.

[mildred]

Yep, it's possible to execute postgresql in a podman container instead, but you lack the easy use and integration. I believe the toolbox is much more easy to use of you can just toolbox enterand then start working. Instead if you have to execute some lengthy podman commands outside of the toolbox first, and redirect ports, you are not as productive.

Also, podman gave me an error when I tried to start the container, but that's not an issue with toolbox.

[PlatonB]

In my case, this issue doesn't allow to run MongoDB:

toolbox enter sudo systemctl enable mongod systemctl status mongod.service

System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down

@debarshiray The issue blocks the migration to non-testing usage of ostree-based distros like Silverblue, Gnome OS, Endless OS. Can I hope this problem will be fixed in the short term?

[PlatonB]

The error is also reproduced in the apx_managed_dnf container in Vanilla OS (@mirkobrombin).

[ericnorris]

I believe this warrants a separate issue, but I found this because I was investigating using toolbox for debugging and wanted systemctl status <service> to work inside of it.

I was able to get this to work by running:

ln -s /run/host/var/run/systemd/system /run/systemd/system

SYSTEMD_IGNORE_CHROOT=1 systemctl status

This issue seems to be more about starting services inside of the toolbox, however. I'm less interested in that, and more interested in making the toolbox as transparent as possible on the host, while still allowing for installing tools, etc. for debugging.

[bam80]

What is the status of this?