brl status fail - /tmp is bound for all strata

[sol-fieldman]

Boot: EFISTUB via dracut and initramfs

Hijacked Strata: Void Linux

FS: BTRFS with subvolumes

Relevant Files: fstab, mtab, brl report output, dmesg

Things I have tried doing:

• repairing all strata
• deleting and reinstalling non-hijack strata
• checking for symbolic links on /tmp and /var/tmp
• checking mtab for bound mounts
• manually tuning /tmp and tmpfs (edit: thanks for pointing out the typo) mount opts in fstab

At this point, my hypothesis is that this error might have something to do with BTRFS and the way subvolumes are handled. Previously had to drop into rescue mode post-hijack to manually create a mountpoint for my snapshots subvolume. Have not come across any day-to-day issues because of this error save for the failed check. Any and all suggestions are welcome.

[paradigm]

brl status' error message here isn't very good. Some background to help you understand what it's complaining about:

At one level of abstraction, pretty much all Bedrock does is redirect requests for resources to the appropriate stratum. This is mostly configuration driven so that users can customize it as need be. The [global] section of /bedrock/etc/bedrock.conf has (effectively) three options for file paths:

• local: The processes will see their own stratum's instance of files at that path. Bedrock ensures things like Debian's apt and Ubuntu's apt each see their own mirrors at /etc/apt/sources.list by making /etc/apt/sources.list local.
• share: All processes should see the same contents at that path. This is how you can have a text editor from one stratum edit a program's source code and a compiler from another compile it; both the editor and compiler see the same file at the same path.
• bind: All processes should see the same contents at that path initially, but any newly created mount points within this file path are local. Bedrock mostly (ab)uses this to avoid recursion concerns when setting up the /bedrock directory.

Bedrock defaults to /tmp being configured to be shared, but the self-check code sees/tmp set up to be bound instead. So long as you don't mount anything in there, there's no effective difference. If for whatever reason you mount a DVD to /tmp/dvd, that dvd will only be visible at that path to processes from whatever stratum ran the mount system call.

I recommend checking that all strata do see the same stuff in /tmp, e.g. by creating files there and running strat <stratum> ls /tmp for all your strata.

• If they do see the same thing, this is probably harmless. My guess you're not actively mounting things within /tmp and so the bind vs shared distinction isn't actually a problem for you. You might be able to edit /bedrock/etc/bedrock.conf's [global] section to move /tmp from share to bind as a way to make brl status happy.
• If they see different files in /tmp, that might actually be a problem; some programs do communicate via shared files within /tmp. Also, per your fstab, you probably want tmp to be a tmpfs, and if you're seeing different things there that might not be the case for all strata.

It's not immediately obvious to me why this issue is occuring. Your guess that it has to do with subvolumes is plausible, as Bedrock isn't currently subvolume-aware; that's on the to-do list for 0.8.x. That said, Bedrock isn't erroring about /home which you've configured to be a subvolume while /tmp isn't actually configured to be one; the causal chain here isn't straight forward. The fact /tmp is a tmpfs is interesting as a non-default but it isn't obvious to me why that'd be a problem, either, as Bedrock is comfortable with things like /dev/shm and /bedrock/run being tmpfs mounts without any issues.

I can try to reproduce this when time allows, but it may be a bit; I'm getting increasingly backlogged with other issues/requests, and since this doesn't appear to be an actual blocker for functionality I can't justify prioritizing over other items.

If you want to debug it yourself, the key thing to know about is the shared:<number> flag in /proc/1/mountinfo. That's how the Linux kernel thinks about this (and why brl status and bedrock.conf call it "shared"). Bedrock wants all instances of share items like /tmp mount point to have the same shared:<number> number. /bedrock/libexec/brl-repair and /bedrock/libexec/brl-status are both shell scripts you can probably work your way through if you know shell and have patience. They've got a bit of tech-debt and aren't the cleanest; 0.8.0 will be a complete re-write of this area.

[sol-fieldman]

Solved it (manually). Was not bedrock.

This turned out to be a dracut/initramfs issue. for anyone looking for a solution for the same problem:

1. Check /proc/1/mountinfo for instances of the bound directory, if you see a duplicate mount, move on to step 2
2. Redo the config of your bootloader install (in my case, EFISTUB via dracut/initramfs) using hijacked system
3. check for any redundant files pertaining to /proc, fstab, or mtab
4. quietly facepalm, bookmark the manpages for your boot method of choice.