Using services from other strata, a report of trying to do it with an overlay filesystem

I've tried some things to make services work across strata, and while it is not very satisfactory, i was able to do it to some degree.

Using overlayfs (has been built into the mainline kernel for some time) I mounted a directory that contained systemd units from another stratum over the current stratum's service directory. This made it so that current stratum's systemd would see these services, however the paths in them are still global. How to solve this isn't exactly clear, but i've had success with patching services by substituting ExecStart=something with ExecStart=/bedrock/bin/strata -r (stratum) something. However, some systemd "security features", namely SystemCallArchitecture, MemoryDenyWriteExecute and NoNewPreviledges make this fail to work unless disabled. This way, I was able to start the syncthing@.unit from another stratum. Another disappointing thing is that overlayfs has to be remounted every time something in the other strata changes, though it is pretty fast to do. This may not be the way forward, especially since it requires patching services to work, but I thought it would be good to place this here to show what has been tried.

Using overlayfs (has been built into the mainline kernel for some time) I mounted a directory that contained systemd units from another stratum over the current stratum's service directory.

If I'm following correctly, you're unioning /usr/lib/systemd/system, right? My concern around using overlayfs as I think you are here is potential package manager conflicts. Consider something like:

Bedrock system has bedrock, debian, and arch strata, where debian is the init stratum.
User installs nginx from arch, which through overlayfs is made available to the debian stratum at what debian sees as /usr/lib/systemd/system/nginx.service
User tells debian's apt to install nginx, which tries to create /usr/lib/systemd/system/nginx.service and throws an error which may surprise users who were not expecting it.

I don't think there's a way to cleanly and reliably get overlayfs to make the file visible to to some processes but not others.

This is fine for personal use/experimentation, but I don't think it's good for a general Bedrock solution in the long run.

Other potential problem is that some init systems do not differentiate between services that are (1) available but disabled by default and (2) services that are available and expected to be run on boot. If we're working toward a general solution for multiple init systems (e.g. runit, openrc, sysv, etc) this strategy would result in a lot of things starting at boot a user may not be interested in. Users don't have complete, obvious control over this as some things get pulled in as dependencies.

My thought here is to make cross-stratum init configuration available at some Bedrock specific location (likely something like /bedrock/cross/init) and have users manually symlink whatever services they're interested in. This should resolve the surprise conflict issue, as users should remember they manually opted in for the given service, as well as concerns around pulling in services they didn't want.

This made it so that current stratum's systemd would see these services, however the paths in them are still global.

Not to be pedantic, but I think you mean "local" here rather than "global." If they're "global" we're already good. The goal you have in mind here is likely to make them "cross."

How to solve this isn't exactly clear, but i've had success with patching services by substituting ExecStart=something with ExecStart=/bedrock/bin/strata -r (stratum) something.

This is exactly what's expected. In fact, it's also how Bedrock automatically handles related issues. Read the comments in /bedrock/etc/bedrock.conf's [cross] and [cross-ini] sections, then look at the Exec lines in files in /bedrock/cross/applications.

My long term thought here is to make something like a [cross-init-config] section that does something similar to make init system configuration be suitable for cross stratum usage. This would be what gets the symlink I mentioned earlier. If we do it right, it could translate not only systemd to systemd but between all notable init systems, which I think would get applause from a large section of Bedrock's audience.

However, some systemd "security features", namely SystemCallArchitecture, MemoryDenyWriteExecute and NoNewPreviledges make this fail to work unless disabled

My current thoughts here are to group such systemd fields into one of the following categories:

Things crossfs can translate to be cross-stratum. Things like ExecStart to which crossfs could prepend /bedrock/bin/strat <stratum> if going to systemd, or could create a shell script wrapper which calls things accordingly for runit, etc.
Likely works cross-stratum as-is across systemd instances, but we aren't translating across init systems. Crossfs just forwards these along as-is from systemd to systemd, and drops them when going from systemd to runit. Systemd dependency stuff is the obvious example here; runit uses a very different dependency model.
Things which we can't make cross-stratum, like NoNewPrivileges that you mentioned. Here we just have to drop them, I guess. Any documentation which mentions symlinking from crossfs into where the init system expects the configuration should mention this possibility and ask users to read what they're enabling and understand what they're doing. This is another advantage to requiring users manually symlink things: they're opt-in to the security feature loss and can't reasonably claim surprise about it.

This way, I was able to start the syncthing@.unit from another stratum

Nice!

Another disappointing thing is that overlayfs has to be remounted every time something in the other strata changes, though it is pretty fast to do.

This is another thing Bedrock's crossfs solves. It populates things on-the-fly. Once files are available in a given stratum, their cross-stratum equivalent are immediately available in crossfs.

This may not be the way forward, especially since it requires patching services to work

I think altering init configuration to be cross-stratum is going to be necessary, but I expect we can automate it, at least for simple cases.

User installs nginx from arch, which through overlayfs is made available to the debian stratum at what debian sees as /usr/lib/systemd/system/nginx.service User tells debian's apt to install nginx, which tries to create /usr/lib/systemd/system/nginx.service and throws an error which may surprise users who were not expecting it.

overlayfs will prioritize files in the upper dir, therefore no files from the main stratum should be replaced, they will only be placed there if they don't exist in the main one. Though indeed it will not isolate services from multiple strata from interacting with each other

If we're working toward a general solution for multiple init systems

I was thinking about at least allowing to unite services for at least one system if the strata has the same one. Making them all work together would require implementing a supervisor that replaces the system supervisor and proxies requests to do things to the child supervisor, at least that is how I think it would be, this may somehow be achived by putting xattrs on services from other strata and replacing paths to these files when the child init tries to access them, as well as paths to binaries they try to access. Though

My thought here is to make cross-stratum init configuration available at some Bedrock specific location (likely something like /bedrock/cross/init) and have users manually symlink whatever services they're interested in. This should resolve the surprise conflict issue, as users should remember they manually opted in for the given service, as well as concerns around pulling in services they didn't want.

may be a better idea.

Not to be pedantic, but I think you mean "local" here rather than "global." If they're "global" we're already good. The goal you have in mind here is likely to make them "cross."

You are right, what I meant is that they tried to access the path in the main stratum instead of the right one.

For the rest of your post I can't add much, as it all seems to be a fairly good idea, though this still means an init daemon or some kind of wrapper for services will be needed.

As for

I think altering init configuration to be cross-stratum is going to be necessary, but I expect we can automate it, at least for simple cases.

I was able to automatically patch services on ubuntu with a simple post-install hook, but that will probably not be that easy with other package managers and if there is a different system for starting services / a different init daemon to sit on top be implemented, it may not be necessary. I've been thinking about it for a few days and things seem to be hard to do without somehow either patching the services or overriding/patching the init daemons.

User installs nginx from arch, which through overlayfs is made available to the debian stratum at what debian sees as /usr/lib/systemd/system/nginx.service User tells debian's apt to install nginx, which tries to create /usr/lib/systemd/system/nginx.service and throws an error which may surprise users who were not expecting it.

overlayfs will prioritize files in the upper dir, therefore no files from the main stratum should be replaced, they will only be placed there if they don't exist in the main one.

If you're stating this as a resolution to the problem I'm illustrating, I don't see how it resolves things at all. You'll still end up with a package manager error about a preexisting file which could easily surprise users.

If you're not stating this as a resolution to the problem I'm illustrating, I'm even more lost as to what this is intended to express.

Though indeed it will not isolate services from multiple strata from interacting with each other

I don't know how service isolation came up here; this reads to me like a non-sequitur.

If we're working toward a general solution for multiple init systems

I was thinking about at least allowing to unite services for at least one system if the strata has the same one.

In principle, that's a reasonable feature to pursue if it's easier/faster than a more general one until a more general one is available.

It's not clear to me that it's meaningfully easier or faster than the general approach I'm proposing.

Making them all work together would require implementing a supervisor that replaces the system supervisor and proxies requests to do things to the child supervisor, at least that is how I think it would be, this may somehow be achived by putting xattrs on services from other strata and replacing paths to these files when the child init tries to access them, as well as paths to binaries they try to access. Though

I think I follow your broad plan to make a supervisor that wraps per-stratum supervisors, but I don't follow:

Why you feel this is required given I just laid out an alternative which doesn't require it.
How xattrs come into play

I see how broadly something like this could be done if we isolate the per-stratum inits, but that defeats the whole point of Bedrock. I don't see how to do something like this without isolating them.

My thought here is to make cross-stratum init configuration available at some Bedrock specific location (likely something like /bedrock/cross/init) and have users manually symlink whatever services they're interested in. This should resolve the surprise conflict issue, as users should remember they manually opted in for the given service, as well as concerns around pulling in services they didn't want.

may be a better idea.

:+1:

For the rest of your post I can't add much, as it all seems to be a fairly good idea, though this still means an init daemon or some kind of wrapper for services will be needed.

I don't follow how my proposal requires an init daemon or wrapper for simple services.

I do see how something like systemd-shim is needed if we're going to run services that have a hard dependency systemd with non-systemd inits, if that's what you mean. AFAIK systemd-shim is dead, and I don't personally plan on reviving it, but if someone else did that'd be useful here.

As for

I think altering init configuration to be cross-stratum is going to be necessary, but I expect we can automate it, at least for simple cases.

I was able to automatically patch services on ubuntu with a simple post-install hook, but that will probably not be that easy with other package managers and if there is a different system for starting services / a different init daemon to sit on top be implemented, it may not be necessary. I've been thinking about it for a few days and things seem to be hard to do without somehow either patching the services or overriding/patching the init daemons.

Most Bedrock efforts try to use as generic a solution as possible. Package manager hooks are much more project-specific and don't scale very well; they're not an avenue I want to go down if I can avoid it.

I also dislike patching stratum local files. This upsets package managers and makes the system even harder to administrate. Leave package manager owned files to their package managers.

I'll rephrase my proposal, in case I wasn't sufficiently clear.

First, background:

One of the main strategies Bedrock currently uses is crossfs, a filesystem mounted at /bedrock/cross. It translates various resources from stratum-local copies to something that works cross-stratum. Bedrock uses this for binary executables, fonts, .desktop entries, etc. The reason this is done in the Bedrock specific /bedrock/cross location, instead of a stratum local location, is that it doesn't upset package managers. They can all do their own thing in in their respective stratum. Bedrock configures resource consumers to look at /bedrock/cross. Things like altering $PATH to teach bash to look at /bedrock/cross/bin. Bedrock has been doing this broad strategy for years. It's fairly well proven at this point.

You noticed, correctly, that init configuration does not current work cross-stratum as is. You can't just copy/paste one stratum's init configuration into another stratum's init's configuration location and expect it to work. You also noticed that you can tweak init configuration to work cross-stratum, such as by injecting strat into the right location. It's not clear to me if you picked this up from reading the documentation or if you figured it out independently.

My proposal:

Use crossfs to do these translations - things like injecting strat - just as it already does for other resources.

The other half of the puzzle is getting inits to know about these service configuration items in /bedrock/cross. I don't have a good, just-works solution here. The obvious ones involve issues such as:

Making the cross-stratum configuration available in the typical init config location. However, this will upset package managers if you install multiple instances of the same package, which is a completely reasonable thing to do. Users are likely to be negatively surprised by this.
- It could be possible to resolve this with a custom filesystem that detects service managers vs package managers and only shows one the cross-stratum files but not the other. While I haven't completely ruled this out, I'm not super fond of it.
Launching services users didn't want to launch, because they were pulled in as dependencies from some other operation.

My proposal here is to have users manually symlink cross-stratum init configuration items from /bedrock/cross to the appropriate location. It's not quite as just-works as I'd like, but it resolves both of the above concerns.

Downsides to my proposal:

As previously mentioned, users have to manually opt-in to cross-stratum services by symlinking them in.
- This is a trade-off I feel is worthwhile, but I'm open to ideas which resolve this without introducing new problems.
Basic systemd things and runit things should be easy. Fancy systemd things and other init systems like OpenRC might be harder to parse and generate. It's not clear how well this will scale.
- This is a trade-off I feel is worthwhile, but I'm open to ideas which resolve this without introducing new problems.

Upsides to my proposal:

Uses a preexisting, well proven Bedrock system.
No upsetting package managers
No new processes/wrappers/daemons
No additional system resource utilization

Sorry, I guess I haven't been reading it properly. I think i get more or less what you mean now. This requires extending crossfs to understand these service files, right? With systemd, just parsing and replacing things would probably work fine, but for openrc scripts i'm not quite sure.. There are some special variables OpenRC uses for simple cases, but otherwise it's all just a shell script where an executable can be used anywhere.

I've tried reading crossfs code but it's a bit too hard for me to understand just from glancing. Perhaps I'll try to work out some initial code for what you're proposing, at least to get to know the filesystem better.

This requires extending crossfs to understand these service files, right?

That's right.

With systemd, just parsing and replacing things would probably work fine, but for openrc scripts i'm not quite sure.. There are some special variables OpenRC uses for simple cases, but otherwise it's all just a shell script where an executable can be used anywhere.

My current thought is to do something like:

ExecStart = strat <openrc-stratum> sh -c '. <path-to-shell-library> && . <path-to-openrc-file> && start'

where the shell library contains OpenRC specific functions like ebegin and eend. If these are located somewhere in the OpenRC providing stratum, we could just source that; otherwise, Bedrock would make its own stub for those.

Going in the other direction, it'd be something like:

#!/sbin/openrc-run

start() {
    ebegin "Starting <path-to-unit-file>"
    strat <systemd-stratum> <unit-file-ExecStart-line>
    eend $?
}

However, I don't know OpenRC well at all, and I could be underestimating the difficulty.

Runit should be the easiest here; they're just single run executables. Turning those into systemd unit files would be something like:

ExecStart = strat <runit-stratum> <path-to-run-file>

and going the other way would just be

#!/bin/sh
strat <systemd-stratum> <unit-file-ExecStart-line>

There may be additional work needed to do to handle things like whether the service daemonizes or not. runit has fghack, systemd has Type=, etc.

My expectation is that an initial release here only supports the simplest of services. We'll then add init system feature translation over time.

I've tried reading crossfs code but it's a bit too hard for me to understand just from glancing. Perhaps I'll try to work out some initial code for what you're proposing, at least to get to know the filesystem better.

What I wrote above was an explanation of where I'm thinking Bedrock goes, and not necessarily asking you to write this. crossfs is probably the hardest part of Bedrock's code base to understand, at least in part due to tweaking over time leading to tech debt rather than because it's optimal. If you want to pursue it, I'm A-okay with that, but if you don't that's fine to. I'm happy to do it once more pressing things are done. That having been said, it'll be a long while before I get to it.

If you are pursuing it, my thought is to:

Create some C struct that represents a generic service.
Create a function per init system to read the init system's configuration file and convert it into the aforementioned struct
Create a function per init system to generate an init system's configuration file from the struct

Having this generic intermediate representation means we just have to write (2 x init-system-count) functions instead of (init-system-count^2) functions.

If you have your heart set on cross-stratum init work, I'm absolutely happy to receive assistance there. However, if you're just looking to contribute to Bedrock in general and you know C, there are two other tasks that I feel are a much higher priority than this one:

GRUB generates a bad grub.cfg on Bedrock when using btrfs and/or zfs. I think it's a bug in grub-mkrelpath which assumes /boot is only mounted into the filesystem tree once, rather than multiple times. This is probably the single biggest limitation for potential Bedrock users right now. If you can upstream a fix to GRUB it would be huge for Bedrock.
Bedrock systems running BSD-style SysV freeze on shutdown. I think the issue here is another Bedrock filesystem, etcfs, crashes on SIGTERM instead of unmounts properly. BSD-style SysV init systems read from /etc after killing etcfs, but because it crashed they can't, and the system freezes. I don't know if the big here is in Bedrock's etcfs code or the libfuse library it's based around. There are people who have been politely hanging around the Bedrock community for years waiting for this to be fixed. This would also unblock BSD-style SysV for cross-stratum init configuration stuff.

If you are interested in pursuing either of those but aren't sure how to reproduce the issue, I can provide more details.

That having been said, Bedrock is all volunteer work: feel free to work on whatever task you have the most interest in, if any.

I've drafted a base for what was discussed in #192. For now it just does more or less what I was doing by hand, except the user has to symlink things to their actual service directory to use them.

I don't see value in going this route.

If you just want to add strat in systemd's unit files files, you don't need to touch crossfs.c at all; you can just

usr/lib/systemd = /usr/lib/systemd

under

[cross-ini]

and you're good to go.

Otherwise, if you want to set up infrastructure to handle things like selectively ignoring some fields (e.g. NoNewPreviledges) or translate between other init systems then merging the logic with the generic ini handling logic won't scale at all.

This is just a start for the rest of the things. I'll be working on the universal format for services next. The PR is far from done, hence it is a draft.

Gotcha. Sounds good!

bedrocklinux / bedrocklinux-userland

Using services from other strata, a report of trying to do it with an overlay filesystem #190