Initial implementation for service sharing

[vaartis]

This is a very basic implementation of service sharing between strata, it simply forwards systemd services to /bedrock/cross/services/ and replaces the Start commands in them with strat-using ones. Somehow, the security features of "systemd" seem to not bother it now, even though previously in my testing they did. After symlinking a service from cross to /etc/systemd/ it works in the main stratum. The plan, as discussed in #190, is to have services from different init systems translated to the other systems via a universal service format on the fly.

I wanted to open a PR early so the progress could be followed easily.

[vaartis]

There is now a foundation for cross-stratum service translation. For now, just the most simple translation is implemented. That is, runit to systemd. Services are created on the fly and stored in memory cache, served from there when needed and updated if the original service file changes. As before, these need to be linked to the init stratum actual service directory to be used, but it is currently already possible to run the simples runit services with systemd.

[vaartis]

I took a bit of a liberty and separated things to several files to keep it a bit easier for me to work with. Most of #defines and some structs I moved to definitions.h, and everything that concerns services is in service_generation.h.

As for services, I have implemented generation of most things runit can do, that is start (of course), finish (something to do after the service is stopped) and conf (a file with environmental variables read before the service starts). runit also has a concept of "check", healthchecks, however systemd has no easy way of running arbitary scripts as healthcheck so I omited that moment from the systemd generator (the value is still read and saved). With that, most runit services should be able to run fine. The next thing I will work on will probably be OpenRC translation. The last thing that will be left is translation to init systems that are not systemd, as it's a bit harder to work on them when it's not a native init system of my distro, though they can probably be set up to run as non-pid-1 processes for testing, so that should not be a big problem.

[vaartis]

After a bit of digging with openrc it seems like the way to run it is:

1. touch (stratum)/run/openrc/softlevel, otherwise it refuses to run
2. touch (stratum)/var/run/openrc/started/networking (and possibly other system things) so it wouldn't try touching the actual system
3. stratum -r (stratum) (stratum)/sbin/openrc-run (full path to service file) start/stop/status

Hopefully this will work well enough for most things.

[paradigm]

Breaking it up into multiple files is fine. That's something I probably should have been doing already. When this effort is done (so we don't fight over merging conflicts) I might refactor more into separate files.

You're clearly doing this with an eye towards maintaining the preexisting style. Things like using using PATH_MAX as the semi-arbitrary string length constant, heavy enum usage, etc. I greatly appreciate that!

I haven't had time to what you've done so far in depth, but it looks like you're caching generated services so you don't have to regenerate on a re-read. Not a bad idea in general, but I'd prefer that be left out of this effort and PR:

• If we do caching, it makes more sense to do it crossfs wide instead of just for service management, in which case it makes more sense as a separate effort and PR.
• Some users run Bedrock on systems with relatively little RAM, and for them the caching trade-off could be a deal breaker. If we do something like this, we'd need user-facing configuration to control things like how much memory crossfs should use for caching. Maybe also logic to check the system's free RAM and voluntarily drop cache automatically. This in turn requires logic to track how much is cached and some algorithm to figure out what toss (maybe least recently used?). This isn't to say it won't ever be a good idea, but doing it right will be a non-trivial effort, and in that case it makes more sense to make it a separate effort from this one.
• The overhead for FUSE itself usually dominates performance concerns, meaning this kind of caching effort probably doesn't give us much benefit for the amount of effort it will take to do and do right. I'd rather limited developer resources go to more pressing things.
• There are other things we could which would have a larger impact on crossfs performance, like leveraging FUSE's readdirplus concept. Last time I looked into this it was broken. I poked at libfuse's code and found the issue, but didn't have time to pursue it. If you're interested in crossfs performance improvements, I could elaborate on my findings and we could pursue that in the future (again as separate from the cross service effort).

[vaartis]

Sorry for taking a bit too long and not responding with anything. I understand your concerns about caching and I have removed it. Maybe it could be done sometime in the future.

As for OpenRC implementation, it is a bit tricky.. it's more complex than sv and it seems to be easier to just use openrc-run directly, however for that to work softlevel file has to exist and the openrc command has to be run at least once so it could create /var/run/openrc stuff. Otherwise, setting a specific PIDFile for systemd and starting openrc-run seems to work. However, services may specifiy their own pidfiles which would create problems with the current implementation of just using the pidfile created by start-stop-daemon.

The next open question now is how to handle dependencies.

[paradigm]

Sorry for taking a bit too long and not responding with anything.

No worries at all! Bedrock is entirely volunteer work, and I fully understand people have other things going on in their lives. No rush on anything here.

I didn't expect to get to this myself in 2020, or see anyone else volunteer to do so. We're way ahead of where I expected to be on this topic.

As for OpenRC implementation, it is a bit tricky.. it's more complex than sv and it seems to be easier to just use openrc-run directly, however for that to work softlevel file has to exist and the openrc command has to be run at least once so it could create /var/run/openrc stuff. Otherwise, setting a specific PIDFile for systemd and starting openrc-run seems to work. However, services may specifiy their own pidfiles which would create problems with the current implementation of just using the pidfile created by start-stop-daemon.

As long as we're careful to document it, I'm perfectly okay with the first release of this having caveats like it failing to work with OpenRC that create their own pidfiles. We can slowly improve these things over time, especially as they get attention from people who know the given init system well.

The next open question now is how to handle dependencies.

I've not thought this through well at all. Feel free to disagree here or propose alternatives. That having been said, here are my current thoughts:

• I think many service managers have some general, default runlevel/target/stage/etc concept. Basic, common dependencies like /proc being mounted and networking being available are provided at that point. If we can't figure out something better, this is probably a reasonable default choice. For example, I think systemd's concept here would be multi-user.target and OpenRC's would be default. I think runit's name for this might be Stage 2, but for our purposes the name doesn't matter as it's the only runit stage services are managed.
• I don't think we can feasibly automate translating dependencies across init system types in general due to different names and conventions. For example, I don't think we can get systemd to understand an OpenRC service's dependencies correctly. For these scenarios, we should always use the above default runlevel target thing.
• I think we might be able to automate translating dependencies across the same init system type due to shared conventions. For example, we might be able to get Arch's systemd to understand Debian's systemd service dependencies. We might be able to get Gentoo's OpenRC to understand an Alpine's OpenRC service dependencies. However, I'm not completely sure about this. If we do this, users might have to manually symlink in missing dependencies as well. It's not ideal, but until we have a better solution I think that's acceptable. We can mention systemd and OpenRC error messages about missing dependencies and how to handle them in the associated documentation.

[paradigm]

I thought of two other open issues we need to figure out:

---

(1) All of the consumers of existing crossfs resources can be configured to look at /bedrock/cross. However, the service managers we're considering all read from a single, hard-coded directory location. If we're not careful, symlinking/copying/overlayfs'ing can lead to crossfs recursing problematically.

• If someone symlinks or overlayfs's a service to /bedrock/cross, depending on [cross]/priority settings crossfs may try to read from itself to generate the service, which will end up recursing problematically.
• If someone copies a service from /bedrock/cross, crossfs will recursively apply changes like Exec=/bedrock/bin/strat global /bedrock/bin/strat arch cupsd and create files users shouldn't use.

One way we could work around this is readlink()'ing files we're inputting here and confirming they're not pointing into crossfs before reading from them. It's a performance hit I'd rather have avoided. If you have other ideas I'd be delighted to consider them.

---

(2) It's not clear to me how we should make the bedrock.conf configuration work.

The existing bedrock.conf [cross-*] pattern is:

[cross-<filter>]
<output-path> = <input-path>

which worked great for past resources, but is a bit clumsy here.

Currently, crossfs cannot merge <output-path>s. Something like this won't currently work:

[cross-systemd]
systemd = /usr/lib/systemd/system
openrc = /usr/lib/systemd/system
runit = /usr/lib/systemd/system
[cross-openrc]
systemd = /etc/init.d
openrc = /etc/init.d
runit = /etc/init.d
[cross-runit]
systemd = /etc/sv
openrc = /etc/sv
runit = /etc/sv

That's easily fixed by just having the <filter> imply sub-directories per service type. Something like this:

[cross-service]
service = /usr/lib/systemd/system, /etc/init.d, /etc/sv

would create

/bedrock/cross/service/systemd/...
/bedrock/cross/service/openrc/...
/bedrock/cross/service/runit/...

I think that's close to what you're doing.

The bigger problem is how to figure out what the input is. If we put it in the <filter>, we end up with something like:

[cross-systemd]
service = /usr/lib/systemd/system
[cross-openrc]
service = /etc/init.d
[cross-runit]
service = /etc/sv

However, crossfs does not support multiple identical <output-path> items in bedrock.conf like that. Another option would be to add additional configuration syntax. Something like:

[cross-service]
service = <systemd>/usr/lib/systemd/system, <openrc>/etc/init.d, <runit>/etc/sv

However, there we run into the possible issue of different distros having the different service managers at the same path. We might in the future run into something like:

[cross-service]
service = <systemd>/usr/lib/systemd/system, <openrc>/etc/init.d, <runit>/etc/sv, <sysv>/etc/init.d

in which case crossfs has to figure out if a given stratum is using OpenRC or SysV. In that case, it gives us little over just always detecting the input type automatically.

We could have crossfs hard-code assumptions about what service type is at what input file path. I think this might be what you're doing now. In that case, though, things fall apart if the user ever tries another path, which wouldn't be unreasonable of them, or if we find something like multiple inits that use the same path (which might already be the case with SysV and OpenRC).

The last option I have in mind is to have crossfs automatically detect the service type from what it's reading when it reads inputs. Something like:

• If the item is a directory, just pass through a directory.
• If the filename is run, it's runit.
• If the item ends in .service, it's systemd
• If the item's first line contains openrc-run, it's OpenRC.
• Otherwise, SysV, maybe?

This adds performance overhead I'm not super happy about, and has the potential to mis-detect something.

I'm not sure which, if any, of these I like. I'm certainly open to ideas here.

[vaartis]

Sorry for lack of any.. anything. The situation in my country of residence is uneasy right now, and the internet connection is unstable, so I have to pause the work for an undetermined amount of time. Hopefully I can return to this soon.

[paradigm]

I absolutely understand. No rush on this effort. Take care of yourself.