src/init/init: There should be an option to enable an 'fsck /dev/root'

[fogti]

There should be an option to enable an 'fsck -p /dev/root' (or something with the same result), called before / is re-mounted read-write. It's possible to work around this using an initrd, but it should be possible to have the root-fs fsck'ed even w/o initrd (e.g. Gentoo normally works well w/o an initrd).

This should be optional (because an initrd could be in place, which setups '/') and it should be checked that '/' is mounted read-only, before an attempt to run fsck is made. It must be run inside of src/init/init because later there is no way (without messing with bedrock internals) to remount '/' safely read-only to perform that fsck.

Currently, no fsck on the root device seems to be performed at all.

[fogti]

ok, I tested the patches here once, and the system still boots, but I haven't checked for unwanted side effects. That patch is somehow important because sometimes the eudev daemon from the gentoo strata hangs or gets killed on shutdown and the filesystem then isn't always cleanly unmounted.

[fogti]

ok, fsck '/' didn't work, but parsing the kernel cmdline does. https://github.com/zserik/bedrocklinux-userland/commit/3d3475c5f994b69b17b2b2c620ac8e12009176d8 one might need to add extra filesystem-type fsck helper executables to the bedrock strata (e.g. for ext4 or other, more obscure fstypes) but then it works.

[paradigm]

This is a known issue: https://bedrocklinux.org/0.7/known-issues.html#fsck-root

I realized this was an issue it too late in Poki's development to resolve it before 0.7.0's release, and I have been swamped with other requests and haven't had a chance to seriously investigate alternatives. I am completely in agreement it needs to be done.

I'd rather not add the fsck helpers directly into Bedrock, if we can avoid it. It'll result in requests for Bedrock to add support for every toy filesystem out there, as well as the expectation we maintain it. Ideally we'll get it from other distros, just like we do everything else.

I would expect we can get it from another stratum by setting up some mounts and chrooting in. The specifics may take some experimentation, but in theory I don't see why it shouldn't be possible.

One concern is which stratum should provide it. If nothing else, we could require users configure it in bedrock.conf and default it to the hijacked stratum on hijack. However, this would then require users change it should then brl remove hijacked. It'd be neat if we could automate detecting which strata provide the appropriate helper and using that one. Maybe we can read /proc/1/mountinfo to detect the filesystem, then search for fsck.<filesystem> in the $PATH.

Is the kernel cmdline value guaranteed to be there? I wouldn't be surprised if there's some bootloader setup which eschews it. I'm inclined towards some other strategy. Can we get the required value out of /proc/mounts or /proc/$$/mountinfo?

[fogti]

The root= cmdline value is probably / could be missing if we use an initrd or the kernel commandline is hardcoded in the kernel, but should be there otherwise.

If we can't use the hijacked stratum, we could fall back to the selected init stratum (we need to handle the case when we are currently hijacking, separately, the most simple solution in that case would be to skip the fsck then).

[fogti]

Getting the value out of /proc/mounts or mountinfo doesn't work because they both refer to /dev/root, but this symlink isn't there when we need to fsck (and I don't know why).

Gentoo (e.g.) utilizes udev udevadm to get the major/minor numbers of the root device and creates the symlink later while booting. (init script snippet from /etc/init.d/udev-trigger)

dev_root_link()
{
    local RULESDIR=/run/udev/rules.d
    [ -d $RULESDIR ] || mkdir -p $RULESDIR
    eval $(udevadm info --export --export-prefix=ROOT_ --device-id-of-file=/ || true)
    [ "$ROOT_MAJOR" -a "$ROOT_MINOR" ] || return 0

    # btrfs filesystems have bogus major/minor numbers
    [ "$ROOT_MAJOR" != 0 ] || return 0

    echo 'ACTION=="add|change", SUBSYSTEM=="block", ENV{MAJOR}=="'$ROOT_MAJOR'", ENV{MINOR}=="'$ROOT_MINOR'", SYMLINK+="root"' > $RULESDIR/61-dev-root-link.rules
    return 0
}

[nift4]

@zserik We don't have udev so early.

[nift4]

@paradigm In the LVM patch we build util-linux which includes fsck (if enabled, currently it isn't) but the filesystem plugins needed to be built external.

[fogti]

@nift4 The current build includes a busybox fsck (I think it supports ext2?). I know we don't have udev so early. It would be interesting if there is a way to emulate the behavoir of the udev rule above without udev.

[nift4]

@zserik ln -s $ROOTDEV /dev/root. Now we only have to get ROOTDEV...

[nift4]

https://bootlin.com/blog/find-root-device/

[nift4]

BusyBox is already included...

[fogti]

@nift4 Thanks.

ok, thus, we need to check if the busybox built by bedrocklinux-userland's Makefile already build the rdev utility or if we need to explicitly enable it in the Makefile, but probably the Makefile should automatically build it if we use it in the init script.

https://github.com/bedrocklinux/bedrocklinux-userland/blob/master/Makefile#L413

[nift4]

Idk ask @paradigm

[paradigm]

zserik's probably is exactly correctly.

The Bedrock Makefile's busybox configuration section:

• Starts with busybox's default config
• Disables applets the default config includes which Bedrock doesn't use
• Then finally explicitly enable anything desired which is not in the default config and disables anything included in the default which was not removed in the above mentioned search

I started with the default config to save developer time from walking through the entirety of the config, learning every bit and figuring out if each is needed. I figured the default would be a good starting place. I wrote the search to automatically trim out unneeded items included in the default config.

I just ran a cd vendor/busybox; make defconfig; grep RDEV .config and saw CONFIG_RDEV=y in the output. Looks like it is in Busybox's default config. Including a reference to rdev in one of Bedrock's shell scripts - such as the meta-init section this github issue is focused around - should be sufficient to keep the unused applet check from removing it. If you try this and find that's not the case - maybe I'm missing something - we'll have to make a new ./set_bb_option line.

[nift4]

./set_bb_option "CONFIG_RDEV" "y" can be used if needed.

[nift4]

@zserik

any ideas how to manage fsck.xxx binaries? (I mean @paradigm said above that he does not want to include fsck in bedrocklinux because everyone wants support for all file systems.)

EDIT: Oh, that mention looks really cool.

[paradigm]

My current preference is users install the helper in some stratum, which Bedrock's meta-init will detect and utilize at the appropriate time. The hijacked stratum will probably come with such a helper meaning it will probably work out of the box. We just need to document installing an appropriate fsck helper if you remove the hijacked stratum, just as we document the need for handing off bootloader control.

My imagined workflow is something like:

• Check some new /berdock/etc/bedrock.conf setting (presumably under [init]) to see if the user wants us to fsck the root device.
  • If not, skip the steps below.
• Detect which device needs to be fsck'd and which helper needs to be used.
  • If we can't find one, print/log a warning.
• Search every /bedrock/etc/bedrock.conf [env-vars]/PATH entry for every stratum for the helper.
• Mount a devpts at /bedrock/strata/<stratum>/dev, and maybe proc and sysfs if needed as well.
• Chroot into the stratum and run the fsck helper on the appropriate device.
• Undo mounts to clean up

I haven't spent any meaningful time thinking about this, being distracted with a plethora of other requirements here, and I could be missing some issue with the above described plan.

[fogti]

I probably won't update the init code regarding fsck anymore, because I switched my rootfs to btrfs, which doesn't use fsck (and allows snapshotting + multiple transparent bind mounts, which make live-backups from brl way easier).

[paradigm]

I'm definitely happy you've got a workable solution for your needs, especially given how long it's taking me to get around to implementing a proper solution here. However, I'm still intent on offering a proper solution for those on ext4 or other filesystems which still need fsck.