FS#3798 - flashing the install image leaves overlay filesystem in undefined state

[openwrt-bot]

slingamn:

I'm using the image builder for 21.02.0-rc1, specifically the one targeting bcm2709.

When I flash an install image, the resulting system image references an overlay filesystem (an ext4), the superblock of which is beyond the end of the install image itself. Therefore, flashing an install image may leave files from a previous overlay filesystem in place. This is counterintuitive, because I expected that flashing the install image would leave the OS in a well-defined state.

As a workaround, I've been appending 64KB of zeroes to the end of the install image, which seems to fix the problem.

Reproduction steps:

1. Build an image, flash it, boot it
2. Create files in the overlay filesystem
3. Remove card and reflash; files in the overlay filesystem remain

Thanks for your time!

[openwrt-bot]

bjonglez:

Can you specify what you mean by "the superblock of which is beyond the end of the install image itself"? How do you see that, and can you reproduce with the ext4 images from https://downloads.openwrt.org/releases/21.02.0-rc1/targets/bcm27xx/bcm2709/ ?

Does the squashfs image work?

[openwrt-bot]

slingamn:

Thanks! I was able to reproduce this with the squashfs factory image:

• https://downloads.openwrt.org/releases/21.02.0-rc1/targets/bcm27xx/bcm2709/openwrt-21.02.0-rc1-bcm27xx-bcm2709-rpi-2-squashfs-factory.img.gz
• SHA256SUM f07d683424fa1e7d642cd0a6a3136d0e56119dc44f8c1c83826961731064723f

using the following steps:

1. Download and flash the image to a SD card
2. Boot the device from the SD card, set a root password, ssh in
3. Create files on the overlayfs, e.g.

root@OpenWrt:~# df -Th Filesystem Type Size Used Available Use% Mounted on /dev/root squashfs 5.3M 5.3M 0 100% /rom tmpfs tmpfs 431.4M 212.0K 431.2M 0% /tmp /dev/loop0 ext4 91.7M 1.5M 83.2M 2% /overlay overlayfs:/overlay overlay 91.7M 1.5M 83.2M 2% / /dev/mmcblk0p1 vfat 63.9M 26.1M 37.7M 41% /boot tmpfs tmpfs 512.0K 0 512.0K 0% /dev root@OpenWrt:~# touch /etc/openwrt_issue_fs_3798 root@OpenWrt:~# touch /usr/bin/openwrt_issue_fs_3798 root@OpenWrt:~# find /overlay/upper/ | grep 3798 /overlay/upper/usr/bin/openwrt_issue_fs_3798 /overlay/upper/etc/openwrt_issue_fs_3798 root@OpenWrt:~# find /overlay/upper/ | grep 3798 | xargs ls -lai 49 -rw-r--r-- 1 root root 0 Apr 18 10:14 /overlay/upper/etc/openwrt_issue_fs_3798 52 -rw-r--r-- 1 root root 0 Apr 18 10:14 /overlay/upper/usr/bin/openwrt_issue_fs_3798

4. poweroff the device
5. Reflash the SD card with the factory image, reinsert, boot up again
6. Root password remains set; files remain in place:

root@OpenWrt:~# find /overlay/upper/ | grep 3798 | xargs ls -lai 49 -rw-r--r-- 1 root root 0 Apr 18 10:14 /overlay/upper/etc/openwrt_issue_fs_3798 52 -rw-r--r-- 1 root root 0 Apr 18 10:14 /overlay/upper/usr/bin/openwrt_issue_fs_3798

What I said about the superblock is at best informed speculation :-) What I'm seeing is that the squashfs image (decompressed) is only 80881906 bytes long, but the total length of the partition table is 360447 sectors, i.e., 184548864 bytes. So there's data referenced in the partitions that is not overwritten by a factory flash. My belief is that the ext4 superblock of /overlay/upper is in this area.

[plan44]

Hi @aparcar,

Did you find a solution for this in the meantime? It seems to me this bug is still unfixed.

I just stumbled over your post while trying to figure out a bcm27xx sysupgrade issue and getting help with in the openwrt-devel mailing list and in the forum.

It seems to me that both have the same origin: the bcm27xx squashfs based setup seems to simply rely on the data following the squashfs to be "not a filesystem" for mount_root, but without actually making sure that's really the case by padding the image.

• in your case, just reflashing the same image did not change any data occupied by the overlay, so the overlay did persist.
• in my case, where the new image was slightly larger, it overwrote a bit of the overlay (f2fs) but just not enough to make mount_root detect "not a filesystem" - instead, it did detect a f2fs but then found it broken, which made the startup after upgrade fail.
• both cases are probably rare - usually the new image is larger and thus kills an existing overlay simply by overwriting enough of it. Maybe because of that, it has not been really recognized as a bug, altough it can lead to a sysupgrade bricking a box.

Unfortunately, I got very little response in the channels I tried (openwrt-devel, forum, maybe there are better places?) so far.

[plan44]

Just found this suggestion which addresses the problem.