search

telent / nixwrt

This repo is dormant : NixWRT is being rewritten as Liminix
https://gti.telent.net/dan/liminix
MIT License
208 stars 10 forks source link

Define a filesystem hierachy for Nix on wireless routers #16

Open phodina opened 3 years ago

phodina commented 3 years ago

The idea to build completely the fimware.bin on a remote builder is definitely a step in the right direction. The embedded device is too constrained to build any derivations in terms of compute power as well as storage. There are usually NOR flash (small 4MiB - 16 MiB) and NAND flash (32 MiB - 256MiB) memory chips. Also the memory layout of each device is unique. To generate the image we can reuse the epic work already done in OpenWrt.

However, in case of the OpenWrt the user has an option to login in and install additional software. A use case might be e.g. tcpdump to debug traffic issues in the network.

Another use case is definitely porting userspace application where a faulty program present in the squashfs could cause the boot to fail leading to reflashing the router through serial line to access the proprietary bootloader and downloading the firmware.bin through TFTP. Repeating this cycle often the flash memory will degrade leading the making a brick, resoldering the memory chip or buying a new router.

To avoid this scenario it we could use EXTROOT approach. These days routers have also USB subsystem and we could leverage that to attach USB Flash drive with e.g. EXT4 filesystem which would be mounted through overlayfs. Therefore we could put the path /nix/store/ on the removable drive solving the issue with storage space, read-only filesystem and flash storage degradation. Nix would be configured to use binary substitutes for derivations. Additionally the extra space can be used to store containers (Docker, Balena, LXD).

To minimalize write operations to the embedded flash memory building derivations should be available only if the external USB flash drive is present.

On the other hand a question arises as we get a newer kernel or anything that's part of the rootfs. It would not recommend to symlink anything that vital on a removable drive. As a result a whole system update would be necessary (flashing a newer version of firmware.bin) by switching into a ramdisk and overwriting the correct partition(s).

Therefore the packages would have to be pinned - e.g. Flakes.

So the conclusion of the idea is to have there rootfs with overlay on the embedded flash memory and another overlayfs on the external flash drive.

telent commented 3 years ago

So my development workflow right now on MIPS is

https://github.com/telent/nixwrt/blob/main/README.md#running-it-from-ram

This works reasonably well for me, but does depend on having a relatively full-featured bootloader - every device I've encountered so far has had U-boot and (mercifully) has included the tftp support. I don't know what the situation usually is on Arm boards.

phodina commented 3 years ago

Sure, loading the image into RAM is also a good idea for development. Though I haven't investigated what kind of bootloader is present on the ARM router - my guess would be some flavour of U-boot.