/overlay/lower/boot is empty

[jakicoll]

Even though the mkinicpio-script uses rbind, /boot is not bound to /overlay/lower/boot. Maybe since at the time it is run, /boot is not mounted?

[jakicoll]

Oh no. I meant to create these issues on my fork as a reminder for myself. Sorry for the noise!

[nils-werner]

Are these actual issues that you want to fix?

[jakicoll]

Yes. :)

[nils-werner]

then why not track the issues here, and post a PR with a fix? :-)

[jakicoll]

Ok, will do. :)

[nils-werner]

Does #5 fix this issue?

[jakicoll]

Just tried it out. Even though /overlay/lower/boot is still empty (I wonder why!), I'd consider it fixed by #5 since after running rwrootfs, /boot is now populated.

[nils-werner]

It may have to do with fstab mounting the FS to /boot which is ontop of the OverlayFS, not into /overlay/lower/boot, which is below it.

You can try changing your fstab to

#
# /etc/fstab: static file system information
#
# <file system> <dir>   <type>  <options>   <dump>  <pass>
/dev/mmcblk0p1  /overlay/lower/boot   vfat    defaults,ro     0       0

[nils-werner]

I just tried it myself and it didn't work. Peculiar, that mounting something to /overlay/lower/boot does not make it show up in /boot.

However if you duplicate the line for both mount locations in /etc/fstab, you will have /boot show up correctly in both places. I've added this detail to the README.md.

Also make sure to use the latest version of rwrootfs, as only this version knows how to remount rw both locations properly.

[jakicoll]

Have you tried that? As I changed my fstab according to the README.md, my Raspi did not finish booting (get pingable). After inserting the SD into my Laptop and removing the entry, it worked again. Sorry I can't provide you with serial console output, since it's a Zero W without headers soldered. See comment below for the cause.

As I tried to mount it by hand, I got this error:

[root@alarmpi alarm]# sudo mount /overlay/lower/boot/
mount: /overlay/lower/boot: /dev/mmcblk0p1 already mounted on /boot.

I'm not sure if it is safe to mount a filesystem twice; I could imagine that you always need to bind in this case, since the filesystem itself won't support concurrent access to it's structure on disk?

So all things considered, I'd rather live with an empty /overlay/lower/boot - as long as /boot is populated and rwrootfs works too.

[jakicoll]

I got it: If you add the additional line in the fstab: /dev/mmcblk0p1 /overlay/lower/boot vfat defaults,ro 0 0 the system will only boot as long as overlayroot is enabled by /boot/cmdline.txt.

So if you attempt to temporariliy disable the protection by removing the overlayroot parameter from /boot/cmdline.txt but keep the additional line in the fstab, your system will not boot up at all, since /overlay/lower/boot does not exist.

As workaround I just created an empty folder /overlay/lower/boot on my root partition. As the overlay is in use, it does not disturb - as it is disabled, it keeps the system booting.

[nils-werner]

Ah, sorry. Yes, I tried it, but at some stage I must have created the directories so it worked even with overlayroot disabled.

OK so what would be a solution to this? Make rwrootfs re-mount /boot to /overlay/lower/boot?

[jakicoll]

I guess a bind as suggested in b08b94ddfb66fc8fb3f1b8b5636125b6f07ae855 line 10 would be the simplest thing to do. Setup will stay easy and rwrootfs will work as expected.

Another possible option would be doing this bind from the initcpio-hook. I tried that yesterday but gave up on it. It's not a top priority to me - important is that rwrootfs provides an editable /boot.

[nils-werner]

But that will leave /boot writeable after you leave rwrootfs. If we mount it to /overlay/lower/boot, you will be able to change it only if you're inside rwrootfs.

Another idea is to pick up the device from /boot and mount it again on /overlay/lower/boot (untested code):

mount "/sys/dev/block/$(mountpoint -d /boot)/dev" /overlay/lower/boot

[jakicoll]

Just tried your snippet:

[root@alarmpi alarm]#  mount "/sys/dev/block/$(mountpoint -d /boot)/dev" /overlay/lower/boot
mount: /overlay/lower/boot: can't read superblock on /dev/loop0.

[jakicoll]

Actually you can bind a ro-mounted filesystem to another dir and then enable write-access only in the other dir:

[root@alarmpi alarm]# mount | grep boot
/dev/mmcblk0p1 on /boot type vfat (ro,noatime,nodiratime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
[root@alarmpi alarm]# touch /boot/test
touch: cannot touch '/boot/test': Read-only file system
[root@alarmpi alarm]# mount --bind /boot/ /overlay/lower/boot/
[root@alarmpi alarm]# touch /overlay/lower/boot/test
touch: cannot touch '/overlay/lower/boot/test': Read-only file system
[root@alarmpi alarm]# mount -o rw,remount /overlay/lower/boot/
[root@alarmpi alarm]# touch /overlay/lower/boot/test
[root@alarmpi alarm]# rm /boot/test 
rm: cannot remove '/boot/test': Read-only file system
[root@alarmpi alarm]# umount /overlay/lower/boot      
[root@alarmpi alarm]# rm /boot/test 
rm: cannot remove '/boot/test': Read-only file system

[nils-werner]

Really? I didn't expect that to work... Do the files you create in the bind dir really end up in the original dir? And are they still there after reboot?

[nils-werner]

Can you try the changes I made in the findmnt branch?

[jakicoll]

Result below. Is there a reason not to use a bind as suggested?

[alarm@alarmpi ~]$ sudo rwrootfs
mount: /overlay/lower/boot: /boot is not a block device.
[root@alarmpi /]# mount
/dev/mmcblk0p2 on / type ext2 (rw,relatime,block_validity,barrier,user_xattr,acl)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=212500k,nr_inodes=53125,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,relatime)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
tmp on /tmp type tmpfs (rw,nosuid,nodev)
run on /run/systemd/resolve/resolv.conf type tmpfs (rw,nosuid,nodev,relatime,mode=755)
[root@alarmpi /]# ls /boot
[root@alarmpi /]# 

[nils-werner]

Result below. Is there a reason not to use a bind as suggested?

Because from what I read about bind mounts, I don't believe that read-write bind-mounting a read-only mountpoint is intended to allow writing to the underlying file-system.

The bind-mount is a mount that exposes a directory structure as another directory structure, it has no knowledge about and nothing to do with the filesystem the structure comes from.

From your tests I cannot deduct if this really really works the way we expect it to work. And even if it does, I am not convinced it is supposed to work, so we would be relying on undocumented and possibly unexpected behaviour.

Of course if there is documented proof that this is supposed to work, I will accept the suggestion. The solution that sounds more fail-safe to me is to simply find the device the mountpoint comes from, and mount it read-write again. That really is documented behaviour and is supposed to work.

[nils-werner]

Regarding the errors you see: I had a typo in rwrootfs, it should work now.

Sorry for these loads of untested code. I don't have a Raspberry Pi here right now and have to guess what works. :-)

[nils-werner]

OK, I stand corrected. --bind is the right way to pick up a mount from somewhere else and mount it again. If you do that, mount even shows the device mounted twice.

I will fix up a solution using --bind now.

[nils-werner]

Check out the master branch. I tested it yesterday on my Raspi and it seemed to work.

[jakicoll]

No worries. I've been happy to help. :) I'll check the master until this weekend.

[jakicoll]

Sorry for the delay. Works like a charm.

Thanks again!