Overlay must be removed after setup

[asmello]

I haven't found any documentation about this (here, in the wiki or elsewhere), but I've discovered that this overlay takes precedence over the overlay created by lbu from the configuration changes after performing the initial setup. So this means that despite running lbu commit -d, many settings (including SSH and user passwords) won't be applied on reboot. Deleting the overlay from the boot filesystem fixes the problem.

For context, I've been following the Raspberry Pi guide. This may be specific to the Raspberry Pi images. It could also be because I'm persisting to a separate partition (mmcblk0p2), and the boot partition always takes precedence for overlay application. Regardless of the cause, I believe the instructions in this repository should be updated to avoid confusion.

[macmpi]

Hi, thanks for your feedback. I also use this overlay a lot on Pi devices and never encountered this issue, but indeed in diskless setup I always use boot partition to store overlay (set with setup_alpine): in that standard situation, first lbu commit -d command will properly modify original apkovl file.

To understand better what's going-on I looked more closely to initramfs which detects boot media (with nlplug-findfs) and elects which apkovl file to use. (There was a mailing-list discussion recently on something pretty much related, maybe you initiated it?)

From the observations, it very much seems file located on boot partition takes precedence by default, which is quite common sense usecase. However, there is room for apkovl= kernel cmdline option to probably provide an alternative, but I have never tested it; you may want to:

apkovl Valid forms include:

• An HTTP, HTTPS or FTP URL to an apkovl.tar.gz file which will be retrieved and applied.
• device_name[:fs_type]:path, where device_name does not include /dev (e.g., sda). fs_type is optional (e.g. ext4). path expresses the path on the device to the apkovl.tar.gz file.
• A relative path, interpreted relative to the root of the alpine_dev.
• If not specified, a file matching *.apkovl.tar.gz is searched for in the root of the ovl_dev. (If more than one exists in the root of a device, all are ignored.)

All this raises a good point: maybe setup-lbu should set some apkovl= kernel cmdline option if user chooses a non-boot partition?... You may want to raise an issue in that repo.

Please let me know how that apkovl cmdline parameter does in your case. Depending on situation, I may add some note on this rather edge case. (I'm not fond of the idea to rename/remove the file, as It would need to be added back each time one intends to reuse the media, just to cover a very rare case. Now, you could do it in automated way with unattended.sh add-on script... of course it would have to remount partition rw first).

[asmello]

Thanks for very quick reply, and the thoughtful investigation. Interesting that you perceive this as a very niche case - in my mind low-power and IoT devices like the Raspberry Pi were a major use case for headless bootstrap. In those constrained environments, where often you have 1GB or less of RAM, there is commonly going to be a need to persist data beyond what tmpfs can hold, and a secondary data partition is naturally going to come up as a workaround. One could technically enlarge and use the boot partition, but that sounds suboptimal since that partition is FAT32 formatted and meant to be read-only.

(There was a mailing-list discussion recently on something pretty much related, maybe you initiated it?)

I didn't, but this shows how I'm not the only one confused about the default behaviour. :)

Good to know about the apkovl= flag, that sounds like a better approach than manually removing the headless overlay. I agree it should be applied automatically by setup-disk, will follow up with an issue there if this works. I think there are other broken features in RPis though, so it doesn't seem like the platform gets much love despite being among the official builds...

Will also see if I can also get a note added to the wiki.

EDIT: Just quickly confirming that setting apkovl=/media/mmcblk0p2/<host>.apkovl.tar.gz in cmdline.txt works. A bit inconvenient to have a hard reference to the hostname there, though.

[asmello]

Will close out this issue, feel free to add a note the readme or not.

[macmpi]

where often you have 1GB or less of RAM, there is commonly going to be a need to persist data beyond what tmpfs can hold, and a secondary data partition is naturally going to come up as a workaround.

There might be a bit of misunderstanding here, and I'm not sure how familiar your are with Alpine install modes: sorry if what I describe below seems obvious to you. Alpine can be installed in 3 modes: diskless (no persistence), data (some data persistence in an ext4 partition), sys (classical all-persistent rootfs, like PiOS or any other distros). The persistence method in Pi wiki is a bit of hackery providing some persistence in a diskless setup (not a Pi-specific thing BTW): it may have some merit in some cases, but in most cases, if you want a persistent system, you would go for plain sys mode install. Once sys mode is installed (through setup-alpine mostly), you won't care about lbu-commit at all anymore, and you'd trash any apkovl file.

If you really intend to use diskless mode (has some merits in some usecases), then I'm not sure I get the point of having apkovl in another partition than boot (even in the frankenstein-diskless-persistent case). But maybe if you explain in more detail what kind of setup you need, I can help in more.

I think there are other broken features in RPis though, so it doesn't seem like the platform gets much love despite being among the official builds...

Please post issues in relevant parts of gitlab repo, and some folks will try to help and fix things. Pi is actively supported, and they are many people contributing on a voluntary bases: your own contributions to improve things will be appreciated.

EDIT: Just quickly confirming that setting apkovl=/media/mmcblk0p2/.apkovl.tar.gz in cmdline.txt works. A bit inconvenient to have a hard reference to the hostname there, though.

glad it could help in some way.

[asmello]

Thanks for the explanation, it's very thoughtful. To follow up, indeed my mental model for how Alpine implements diskless mode was a bit off. It didn't click with me that the /etc contents should all be configuration files and so pretty small - no real downside in keeping them in the boot filesystem. Same goes for for the apk cache, in practice, since those packages will be loaded into tmpfs at boot anyway, so you can only store so much there even if you offload to another partition.

In my defence, I read some ~10 different guides and tutorials online specifically tailored for the RPi, and the majority recommended the very setup I was trying to implement. I think the authors must not have understood what they were doing either...

Worth calling out that I learned (the hard way) that keeping the apk cache and the overlay file in separate partitions is a bad idea too. The boot sequence applies the overlay and runs with it before all partitions have been loaded, so if you have cache files in a secondary partition they may not be available yet by the time they get used when packages are installed. Apparently apk won't even download the packages from scratch either. This causes packages to not persist after reboot, even though the cache files are written to. Unfortunately, when this happens, it's a silent failure, so it's very hard to debug.

Anyway, I ended up keeping things mostly as per the default diskless setup. The only divergence is that I applied a couple of bind mounts to /var/lib and /usr/local/bin, routed from the secondary partition. This will help me keep memory under control even when running k3s. I tried using overlays at first but that doesn't seem to work (doesn't throw any errors either...). Hopefully there won't be a race condition at boot, but I think Alpine doesn't really use these directories itself, and so far it seems fine.

[macmpi]

Yes, it does take a bit of time to digest how Alpine may be setup, and benefits of the various options: quite well thought-out actually. In fact there is very little need for platform-specific customization, but people tend to jump on guns with ready-to-go recipes for their particular platform (been there, done that - has some merit to get running)...to then discover after some afterthought, there might be simpler/more elegant or effective solutions. Wiki needs a bit of cleanup, and that's gradually happening: again any contribution welcome.

bootstrap work in this repo also started as a platform-specific project (on Pi), to then realize it can be crafted generic for all platforms. I just included a (somehow Pi-specific) goody to bootsrap PiZeros as ethernet gadget with just one single USB cable: very convenient.

Happy hacking on Alpine !

[macmpi]

FWIW latest version has few code snippets in sample unattedded.sh, one of which would remove overlay. Of course it can be customized at will.