Orange Pi Zero ramfs

[benhylau]

Is there a reason why raspberrypi2 uses tmpfs and sun8i-h2-plus-orangepi-zero uses ramfs?

I am trying to back the fs with swap memory and it works on Raspberry Pi 3 but not on Orange Pi Zero, possibly due to lack of swap support on ramfs.

[darkdrgn2k]

What is rootfs?
---------------
...
If CONFIG_TMPFS is enabled, rootfs will use tmpfs instead of ramfs by
default.  To force ramfs, add "rootfstype=ramfs" to the kernel command
line.

(source: https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt)

[darkdrgn2k]

I dont have a 512mb working orange pi at the moment

But possibly trying to add extraargs="rootfstype=tmpfs" /boot/armbianEnv.txt might do the trick

[benhylau]

Tried appending extraargs="rootfstype=tmpfs" and also just rootfstype=tmpfs, but backing fs is still ramfs not tmpfs:

root@bloor:~# mount /dev/mmcblk0p1 /mnt
root@bloor:~# cat /mnt/boot/armbianEnv.txt
verbosity=7
console=both
disp_mode=1920x1080p60

# Most boards are expected to have the USB hat, so enable these ports
overlays=sun8i-h3-usbhost0 sun8i-h3-usbhost2 sun8i-h3-usbhost3
rootfstype=tmpfs
root@bloor:~# docker info | grep Filesystem
 Backing Filesystem: ramfs
root@bloor:~# df -a | grep rootfs
rootfs                 0      0         0    - /
rootfs                 0      0         0    - /var/lib/docker/plugins
rootfs                 0      0         0    - /var/lib/docker/overlay2

[hamishcoleman]

The kernel has some non-obvious logic on which filesystem to use for the initrd

[darkdrgn2k]

This patch seems to have broke the boot

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.13.0 (hamish@x230i5) (gcc version 6.3.0 20170516 (Debian 6.3.0-18)) #25 SMP Tue Sep 12 14:03:50 HKT 2017
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=50c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: Xunlong Orange Pi Zero
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 16 MiB at 0x4f000000
[    0.000000] percpu: Embedded 16 pages/cpu @ced62000 s35340 r8192 d22004 u65536
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 64960
[    0.000000] Kernel command line: console=tty1 console=ttyS0,115200 cgroup_enable=memory swapaccount=1 hdmi.audio=EDID:0 disp.screen0_output_mode=1920x1080p60 panic=10 consoleblank=0 enforcing=0 loglevel=7 ubootpart=00000000-01 ubootsource=mmc   sunxi_ve_mem_reserve=0 sunxi_g2d_mem_reserve=0 sunxi_fb_mem_reserve=16
[    0.000000] PID hash table entries: 1024 (order: 0, 4096 bytes)
[    0.000000] Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
[    0.000000] Memory: 186212K/262144K available (7168K kernel code, 386K rwdata, 2428K rodata, 1024K init, 337K bss, 59548K reserved, 16384K cma-reserved, 0K highmem)
[    0.000000] Virtual kernel memory layout:
....
[   37.685594] ehci-platform 1c1b000.usb: USB 2.0 started, EHCI 1.00
[   37.692076] mmc0: new high speed SDHC card at address 59b4
[   37.698837] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002
[   37.705710] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   37.712999] usb usb3: Product: EHCI Host Controller
[   37.717907] usb usb3: Manufacturer: Linux 4.13.0 ehci_hcd
[   37.723353] usb usb3: SerialNumber: 1c1b000.usb
[   37.728629] mmcblk0: mmc0:59b4 SD    7.23 GiB 
[   37.734699] hub 3-0:1.0: USB hub found
[   37.738696]  mmcblk0: p1
[   37.742431] hub 3-0:1.0: 1 port detected
[   37.747766] ohci-platform 1c1b400.usb: Generic Platform OHCI controller
[   37.754528] ohci-platform 1c1b400.usb: new USB bus registered, assigned bus number 4
[   37.762615] ohci-platform 1c1b400.usb: irq 29, io mem 0x01c1b400
[   37.834436] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
[   37.841290] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   37.848557] usb usb4: Product: Generic Platform OHCI controller
[   37.854530] usb usb4: Manufacturer: Linux 4.13.0 ohci_hcd
[   37.859958] usb usb4: SerialNumber: 1c1b400.usb
[   37.865722] hub 4-0:1.0: USB hub found
[   37.869583] hub 4-0:1.0: 1 port detected
[   37.874993] usb_phy_generic usb_phy_generic.0.auto: usb_phy_generic.0.auto supply vcc not found, using dummy regulator
[   37.889190] sunxi-mmc 1c10000.mmc: allocated mmc-pwrseq
[   38.170132] sunxi-mmc 1c10000.mmc: base:0xd1292000 irq:24
[   38.176109] vcc3v0: disabling
[   38.179111] vcc5v0: disabling
[   38.182150] ALSA device list:
[   38.185144]   No soundcards found.
[   38.189556] RAMDISK: Couldn't find valid RAM disk image starting at 0.
[   38.196489] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[   38.204077] Please append a correct "root=" boot option; here are the available partitions:
[   38.212510] 0100            4096 ram0 
[   38.212516]  (driver?)
[   38.218674] 0101            4096 ram1 
[   38.218679]  (driver?)
[   38.224858] 0102            4096 ram2 
[   38.224863]  (driver?)
[   38.231039] 0103            4096 ram3 
[   38.231044]  (driver?)
[   38.237209] b300         7584768 mmcblk0 
[   38.237216]  driver: mmcblk
[   38.244081]   b301         1024000 mmcblk0p1 00000000-01
[   38.244086] 
[   38.250953] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[   38.259266] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.13.0 #25
[   38.265294] Hardware name: Allwinner sun8i Family
[   38.270066] [<c010bd5d>] (unwind_backtrace) from [<c01090c3>] (show_stack+0xb/0xc)
[   38.277696] [<c01090c3>] (show_stack) from [<c070064b>] (dump_stack+0x67/0x74)
[   38.284976] [<c070064b>] (dump_stack) from [<c0118381>] (panic+0xa5/0x1b4)
[   38.291896] [<c0118381>] (panic) from [<c0b00e87>] (mount_block_root+0x117/0x1c2)
[   38.299435] [<c0b00e87>] (mount_block_root) from [<c0b010b3>] (mount_root+0xd5/0xda)
[   38.307233] [<c0b010b3>] (mount_root) from [<c0b011af>] (prepare_namespace+0xf7/0x138)
[   38.315206] [<c0b011af>] (prepare_namespace) from [<c0b00bed>] (kernel_init_freeable+0x1cd/0x1e0)
[   38.324122] [<c0b00bed>] (kernel_init_freeable) from [<c070d38b>] (kernel_init+0x7/0xcc)
[   38.332266] [<c070d38b>] (kernel_init) from [<c0105e71>] (ret_from_fork+0x11/0x20)
[   38.339902] Rebooting in 10 seconds..

[hamishcoleman]

I dont know exactly what you are seeing there, but it works for me (https://gist.github.com/hamishcoleman/d7028018eb1e8f869aa6258d48a1e621)

However, it might be that you are running out of memory:

[ 0.000000] Memory: 186212K/262144K available (7168K kernel code, 386K rwdata, 2428K rodata, 1024K init, 337K bss, 59548K reserved, 16384K cma-reserved, 0K highmem)

Running the ramdisk on devices with only 256Meg is never going to work well.

Just doing some maths: the above line says you have 186M free, and you need to unpack at least 118M of files into that space, leaving 66Meg. Checking the numbers from the other direction: the space used on a booted system for the ramdisk is 175M, leaving even less ram from your 186M.

"RAMDISK: Couldn't find valid RAM disk image starting at 0." is a pretty generic error spat out by the kernel when it runs out of ram.

So, I'd say you are just running out of ram.

[darkdrgn2k]

Strange it worked before (tm) I'm rocking the 256 model

I'll try to revert the merge and test again see if it still bails out.

[darkdrgn2k]

Clones,Checkout, Built and Flashed the following commits Each into a new folder to eliminate make clean error Again this is for output/sun8i-h2-plus-orangepi-zero.img

git checkout 7b8c27007f73a90b88cfa344ae34e3433637025 - Froze on Starting kernel ... git checkout 48a7a6c77c64e77543cf1a28e339586efc0297fc - Froze on Starting kernel ... git checkout 4506b89a6327ffe413ced7cde0fecf9e14d1759b - Froze on Starting kernel ... git checkout 1e0f7fd0adea88f7fbb90ed1d91794c84610f632 - Froze on Starting kernel ... git checkout 0fe72f5f849408ec4da3845d945d4dda1b5ffeb2 - Booted fine

** I noticed uInitrd reported about 4 MB smaller then previous at ~ 38MiB vs ~42MiB

I then cherry picked 04498f088ebc864e391d2e9b2803c4994d677b2f into the working build 0fe72f5f849408ec4da3845d945d4dda1b5ffeb2 git cherry-pick 04498f088ebc864e391d2e9b2803c4994d677b2f same error as above

[   42.162154] ALSA device list:
[   42.165146]   No soundcards found.
[   42.169588] RAMDISK: lzma image found at block 0
[   42.184122] RAMDISK: EOF while reading compressed data
[   42.184135] unexpected EOF
[   42.192075] RAMDISK: EOF while reading compressed data
[   42.192082] unexpected EOF
[   42.202820] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[   42.210438] Please append a correct "root=" boot option; here are the available partitions:
[   42.218845] 0100            4096 ram0
[   42.218850]  (driver?)
[   42.225044] 0101            4096 ram1
[   42.225049]  (driver?)
[   42.231226] 0102            4096 ram2
[   42.231231]  (driver?)
[   42.237387] 0103            4096 ram3
[   42.237392]  (driver?)
[   42.243579] b300         7584768 mmcblk0
[   42.243585]  driver: mmcblk
[   42.250457]   b301         1024000 mmcblk0p1 00000000-01
[   42.250461]
[   42.257310] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[   42.265623] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.13.0 #25
[   42.271652] Hardware name: Allwinner sun8i Family
[   42.276425] [<c010bd5d>] (unwind_backtrace) from [<c01090c3>] (show_stack+0xb/0xc)

[ 42.210438] Please append a correct "root=" boot option; here are the available partitions: seems to indicate a missing variably?

But if you say it works, and it is a RAM issue then maybe defining the root partition as rootfs then trying to copy it to a tmpfs?!?!? Shot in the dark.

Edit: linux/build/linux-armhf/lib/modules/4.13.0/kernel/drivers/media contains ~ 8 mb of drivers mostly tuners (tv, radio, satellite) and such and probably can be removed

[hamishcoleman]

From all your checks of different versions, I think you might benefit from knowing about "git bisect" - give it a try for this kind of hunt.

Have you tested your builds in the qemu_armhf emulator? I encourage you to do so, that provides a simple testbed.

After you see that it works in qemu, try changing the "-m 512" to "-m 256", then try adding these two options one at a tiime "-append 'rootfstype=ramfs'" and "-append 'rootfstype=tmpfs'"

This is basically, an either/or situation - either support booting 256Meg systems /or/ support running tmpfs as root.

Yes, there are lots of low hanging fruit for making more space - if you think 8meg of kernel modules is a lot, then you clearly have not noticed the 11Meg of docs in usr/share/doc - but the minimum ram needed with tmpfs is currently around 380Meg (which ends up calculating as needing to find at least 62Meg to delete in the ramdisk - (380-256)/2 - which is a 50% size reduction and not so easy)

Another alternative is to patch the kernel to parse a rootflags= argument that increases the default tmpfs size from 50% to (say) 80% RAM. But down that path dragons lie - people have been dancing with that idea for some years now (search for "rob landley tmpfs size" to see some of it - its still going on)

[darkdrgn2k]

you might benefit from knowing about "git bisect"

I avoided that as i wanted to keep all the build as clean as possible. I didn't know if the Debian directory link would work properly across bisects. I basically did bisect manually but lucky me, i ended up running through like every commit LOL

qemu_armhf emulator

Ill compile it now and test more

you clearly have not noticed the 11Meg of docs

I did but i forgot about them :+1:

tmpfs

This is where things gets harry. We don't NEED tmpfs for a production node (what mesh orange was supposed to be) Its only because were trying to pack it full of so much software that its bursting at the seams! And now we have a RAM backed root file system backed by SD swap file! I dont think mesh orange was ever meant to be what were trying got make it be (ie running docker)

[hamishcoleman]

yeah, with submodules you need to follow each git bisect with a "git submodule update".

If you want to see just how much can be removed, I have a lot of space reduction rules in the https://github.com/hamishcoleman/debian-rescue-initramfs project (look in the packages.d directory) - this project is slowly being refactored so it can use the debian-minimal-builder as a submodule, which will end up making this "files" removal infrastructure easily available to the mesh-orange repo. I have not done a head-to-head comparison for a while, but I think this can reduce the size by at least 50% - with the caveat that it makes the build more fragile and less like debian.

Tmpfs does bring a lot of benefits beyond just swap-backing - bind mounts, xattrs, quotas, df, less crashy all come to mind. Whereas ramfs is intended to be 'just enough filesystem' to allow an initramfs to start the rest of the system and stop using ramfs. So, it is probably worth having anyway.

I do hope we can address these way-out-of-scope use cases like docker with a "xD" version (x for 'anything but ram') that has a root filesystem on a block device to use instead of the current RD system - but with pretty much the same semantics

[darkdrgn2k]

Qemu fails the same with

root@userPC:~/meshtest1/1/mesh-orange# make -C boards/qemu_armhf test
make: Entering directory '/home/user/meshtest1/1/mesh-orange/boards/qemu_armhf'
make -C /home/user/meshtest1/1/mesh-orange/debian build/debian.stretch.armhf.cpio CONFIG_DEBIAN_ARCH=armhf
make[1]: Entering directory '/home/user/meshtest1/1/mesh-orange/debian'
make[1]: 'build/debian.stretch.armhf.cpio' is up to date.
make[1]: Leaving directory '/home/user/meshtest1/1/mesh-orange/debian'
qemu-system-arm -M virt \
    -m 258 \
    -kernel ../../linux/build/linux-armhf/zImage \
    -initrd build/combined.initrd \
    -netdev type=user,id=e0 -device virtio-net-device,netdev=e0 \
    -nographic
[    0.040401] /cpus/cpu@0 missing clock-frequency property
[    1.989371] dmi: Firmware registration failed.
[   65.078381] /initrd.image: incomplete write (-28 != 43147950)
[   65.605841] dmi-sysfs: dmi entry is absent.
[   65.672357] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[   65.672962] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.13.0 #25
[   65.673338] Hardware name: Generic DT based system
[   65.674982] [<c010bd5d>] (unwind_backtrace) from [<c01090c3>] (show_stack+0xb/0xc)
[   65.675761] [<c01090c3>] (show_stack) from [<c070064b>] (dump_stack+0x67/0x74)
[   65.676503] [<c070064b>] (dump_stack) from [<c0118381>] (panic+0xa5/0x1b4)
[   65.676895] [<c0118381>] (panic) from [<c0b00e87>] (mount_block_root+0x117/0x1c2)
[   65.677389] [<c0b00e87>] (mount_block_root) from [<c0b010b3>] (mount_root+0xd5/0xda)
[   65.677956] [<c0b010b3>] (mount_root) from [<c0b011af>] (prepare_namespace+0xf7/0x138)
[   65.678482] [<c0b011af>] (prepare_namespace) from [<c0b00bed>] (kernel_init_freeable+0x1cd/0x1e0)
[   65.679073] [<c0b00bed>] (kernel_init_freeable) from [<c070d38b>] (kernel_init+0x7/0xcc)
[   65.679600] [<c070d38b>] (kernel_init) from [<c0105e71>] (ret_from_fork+0x11/0x20)
[   65.680785] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

[darkdrgn2k]

However 0fe72f5f849408ec4da3845d945d4dda1b5ffeb2 is the last build that works on 256 mb ram even without this patch.

[hamishcoleman]

I know what the qemu build does - I checked.

Did you try it with the other options I suggested?

As I said before:

Choose one - either support 256Meg RAM systems /or/ use tmpfs - not both.

[benhylau]

I would much prefer tmpfs across all boards since it is what we have on the Raspberry Pi systems and it's working great. ramfs has other issues like how the rootfs size is not properly reported with df, that was the case on the orange pi zero before. Because of the 50% ram allocation I didn't expect 256 MB systems to work, but I don't think it's necessary to support those systems. We should perhaps indicate 512 MB-only on that board?

[benhylau]

Confirmed that this works on the 512 MB Orange Pi Zero and I am able to use swap memory to back the tmpfs. However, without swap backing (which I apply as a conf.d script), the rootfs is > 80% full, and adding any package to the base image would make the device unable to boot up. I am guessing there is no solution to this aside from keeping the image small?

[darkdrgn2k]

Just pointing out that the whole point of a ram disk OS is to not relay on the SD card. With swap backing this probably goes the other way, doing allot of writing to the SD card mostly likely shortening its life dramatically.

I realize the swap backing was a workaround for a very specific use case (Workshop) however if it does not stay only there its something to be mindful of.

I wonder why tmpfs eats up more ram then ramfs. Sadly i no longer had a working 512 board to test on and 256 will no longer boot using tmpfs.

[benhylau]

Closing this since the original issue of using tmpfs consistently across Raspberry and Orange boards is resolved, and I can use swap for all of them.

[hamishcoleman]

@darkdrgn2k tmpfs doesnt use more RAM than ramfs - almost the exact opposite.

tmpfs is a more featureful filesystem, and amongst those features is a quota - to stop people from crashing a system by filling up all the ram on a ramfs. This quota defaults to 50% of the system RAM thus explaining why it runs out of space sooner than the ramfs. Once the system is booted up, the root filesystem can be remounted with a different quota, but there is no kernel option to change the quota until then.

When I get some time, I will look at putting a config option to build images with the ramdisk type of your choice, but until then - it is a simple change for you to add "rootfstype=ramfs" to the kernel boot cmdline.