Closed ThomasKaiser closed 8 years ago
Is anyone working on this already?
Don't think so (I mean something on top of our build script).
The efforts for both variants would be the same since based on my understanding you transfer through sunxi-fel what would be otherwise on a minimal (and pretty small in size) SD card for the 2nd variant.
You mean something like this? https://linux-sunxi.org/Miniroot
Then instead of burning an SD card at the end of image creation you power cycle the board you want to test on before and as soon as image creation is ready sunxi-fel automagically transfers SPL/u-boot to the device which then loads the rest through NFS from the build host.
If we are booting from USB FEL mode to ramdisk (essentially), it's easier to transfer rootfs image via USB too instead of dealing with NFS/SFTP TFTP on host, DHCP/static configuration on target and other network stuff, even though network transfer speed may be faster.
I checked wiki, especially sunxi-fel write speeds, and now I think that loading rootfs from network is better. I'll think about this later.
@zador-blood-stained thanks for the feedback. I would also think network is way faster and it might only affect a few core devs to setup the stuff correctly (quite easy BTW if we take care from within build system already and define the build environment -- eg. 14.04 LTS).
The 1st thing I would try out to get this up and running would be to check if it's possible to split the usual images we create into two parts:
If we support this separation then all sorts of upgrade problems when people chose to put their rootfs on NAND/eMMC/SATA would also be solved? Or am I'm thinking (again) too positive?
Anyway: I'll have a look into this within the next half year anyway since I think it would push Armbian a lot in more professional environments if we would support an equivalent to PXE (using small/crappy SD cards to provide the bootloader) out of the box.
PXE may be easier than you think: http://www.denx.de/wiki/view/DULG/UBootCmdGroupDownload You may only need u-boot, boot script and tftp server. Preparing proper rootfs image actually may be harder.
I have some bad news and some good news. Bad news - it appears that initrd with mainline kernel/u-boot is broken for some reason; trying to boot kernel with initrd results in weird kernel panic. Good news - I was able to boot cubietruck with USB FEL and NFS rootfs in more or less automated way
root@cubietruck:~# findmnt /
TARGET SOURCE FSTYPE OPTIONS
/ 192.168.1.102:/home/zador/armbian/output/cache/sdcard/ nfs rw,relatime
root@cubietruck:~#
@ThomasKaiser Once this is merged, you can try it yourself. Note: It requires sunxi-tools installed and present in $PATH; sunxi-tools package in Ubuntu Wily is outdated; obviously requires device in FEL mode connected to build host. Tested on Ubuntu Wily host and cubietruck target. Build script options (Edit: changed in latest version):
./compile.sh KERNEL_ONLY=no BOARD=cubietruck BRANCH=next PROGRESS_DISPLAY=plain USE_MAINLINE_GOOGLE_MIRROR=yes RELEASE=jessie BUILD_DESKTOP=no EXPERIMENTAL_DEBOOTSTRAP=yes FEL_BOOT=yes COMPRESS_OUTPUTIMAGE=no
Edit: Feel free to fix fsid
in fel_prepare_host
, right now armbian mount is set with fsid=root
Edit 2: Unmounting rootfs looks broken, will try to fix later Appears to be fixed
@zador-blood-stained Wow, looks great. This will help improve especially 1st boot scenarios a lot since time between build and boot is 0. Thank you a lot!
I will then add in a hook to power cycle the board(s) in question -- remotely: currently I'm thinking about moving my build machine to another room since the fans get quite annoying under load. This real PC will be assisted by an Orange Pi PC connected to a few relais and UART adapters on each USB port. The hook will then let the OPi PC remotely switch off the other board(s), cut also power to the UART adapter (using sunxi-pio) since some boards behave strange when not fully power cycled then switch board and UART adapater on and provide a remote serial console (to each of the boards to test -- if I understand correctly we are also able to immediately test 1-n boards after some minor tweaks)?
BTW: I learned yesterday that sunxi-fel also provides a progress bar ('-p').
And if I understand correctly now we already have the foundation to build images useable with ultra small SD cards and rootfs on NFS?
Just FYI: I'll add support for legacy kernels (that use script.bin) later. Also I'll check if sun8i and sunxi-default kernels have required NFS options to use NFS root.
And if I understand correctly now we already have the foundation to build images useable with ultra small SD cards and rootfs on NFS?
The only requirements for NFS rootfs (apart from shared rootfs on NFS server) are:
About tftp - you can have something like this in u-boot script
setenv autoload no
dhcp
setenv tftp_ip 192.168.1.2
tftp ${kernel_addr_r} ${tftp_ip}:zImage
tftp ${fdt_addr_r} ${tftp_ip}:sun7i-a20-cubietruck.dtb
setenv fdt_high ffffffff
setenv bootargs "..."
bootz ${kernel_addr_r} - ${fdt_addr_r}
Edit: BTW, @igorpecovnik, is linux-sunxi-default.config
used somewhere?
Thx, understood. Regarding sun8i there's no need to hurry since Ethernet support for H3 in u-boot isn't ready yet and we couldn't test either.
What about image resizing? _FIXED_IMAGESIZE is mandatory I would suppose?
BTW: If this thing is flying one of the first things I'll try out is to switch to btrfs with activated compression. But more as a testbed for future 'normal' images that make use of NFS.
Thx, understood. Regarding sun8i there's no need to hurry since Ethernet support for H3 in u-boot isn't ready yet and we couldn't test either.
We don't need Ethernet in u-boot if we are loading kernel and script.bin via FEL (Edit: or from SD).
What about image resizing? FIXED_IMAGE_SIZE is mandatory I would suppose?
OK, looks like you didn't use debootstrap-ng before. FIXED_IMAGE_SIZE is needed for F2FS root (as you know f2fs is not resizable ATM), otherwise image size is calculated based on actual rootfs use before creating image file, so image shrinking is not needed at all.
BTW: If this thing is flying one of the first things I'll try out is to switch to btrfs with activated compression. But more as a testbed for future 'normal' images that make use of NFS.
BTRFS on host for rootfs? If there is enough RAM on host, rootfs is kept in tmpfs mount, which is even faster.
Alright, yes Ethernet in u-boot isn't needed (was already dreaming of PXE and a single small SD card image ;) )
Regarding image resizing I was talking about /root/.no_rootfs_resize since otherwise the resize would happen or am I'm missing something?
Regarding image resizing I was talking about /root/.no_rootfs_resize since otherwise the resize would happen or am I'm missing something?
Resize won't happen due to first part of conditional expression "$rootfstype" == "ext4"
Ok, I was missing this part and thought blkid would still output "ext4"
linux-sunxi-default.config is deprecated.
Different but related use case: Create images that work the same way but boot from SD card only containing u-boot and initiate from there loading kernel+rootfs through TFTP/NFS.
I made an attempt to implement this: small (64MB for now) SD card image that contains kernel, u-boot script and .dtb/.bin files + rootfs in .tgz archive for deploying on NFS server. SD card is mounted to /boot, so it should be possible to update kernel as usual.
It needs huge amounts of documentation, but for now comments in code and scripts should be enough for testing. Let's hope I didn't forget anything and didn't break anything in debootstrap-ng.
FYI, I didn't test it yet.
Great, thx, @zador-blood-stained !
Looking forward to testing but had to realise that my build VM has no USB ports and everytime I configure one port the VM crashes. I won't investigate further but reactivate a more powerful physical machine in another room. Might take some time and I'm also planning to go on vacation
Looking forward to testing but had to realise that my build VM has no USB ports and everytime I configure one port the VM crashes.
I just thought - it may be possible to use some kind of usb over ip solution (like USB redirector) with server running on another board, so build host could fel-load one board through another.
Great idea! But it will take some time for me since I've to sort out other stuff before and need physical access anyway.
Short feedback. I tried it out today with an Orange Pi PC powered by my new build host. Two showstoppers:
More feedback later, now at dinner.
the sunxi-fel version I installed was too outdated (I will check whether it would be sufficient to build the most recent version from github)
yes, compiling and installing (make install
or checkinstall
if you like keeping your stuff in packages) should do
I meant building the tools as part of the process. But since we're 5 people on this world that might ever use this... ;)
Next step:
U-Boot SPL 2016.03-rc2-armbian (Mar 13 2016 - 20:54:27)
DRAM: 1024 MiB
Trying to boot from
U-Boot 2016.03-rc2-armbian (Mar 13 2016 - 20:54:27 +0100) Allwinner Technology
CPU: Allwinner H3 (SUN8I)
I2C: ready
DRAM: 1 GiB
MMC: SUNXI SD/MMC: 0
Card did not respond to voltage select!
*** Warning - MMC init failed, using default environment
In: serial
Out: serial
Err: serial
Net: No ethernet found.
starting USB...
No controllers found
Hit any key to stop autoboot: 0
(FEL boot)
## Executing script at 43100000
Kernel image @ 0x42000000 [ 0x000000 - 0x496e48 ]
Starting kernel ...
Error: unrecognized/unsupported machine ID (r1 = 0x00000000).
Available machine support:
ID (hex) NAME
00001029 sun8i
Please check your kernel config and/or bootloader.
Ok, a good reason to cleanup the sun8i hacks here and there and to come up with a clean solution. And in the meantime I test with Banana Pi/Pro instead :)
Will try it next time with individual userpatches/fel-boot.cmd (great, everything already prepared):
setenv nfs_ip FEL_LOCAL_IP
setenv nfs_root FEL_ROOTFS
setenv branch BRANCH
gpio set PL10
gpio set PG11
setenv machid 1029
setenv bootm_boot_mode sec
setenv bootargs "console=tty1 console=ttyS0,115200 root=/dev/nfs nfsroot=${nfs_ip}:${nfs_root} ip=dhcp rw panic=10 consoleblank=0 enforcing=0 loglevel=6 rootflags=noatime,nodiratime"
if test ${branch} != default; then
setenv fdt_high ffffffff
bootz ${kernel_addr_r} - ${fdt_addr_r}
else
bootz ${kernel_addr_r}
fi
Nobody noticed that before, but u-boot installation in debootstrap appears to be broken (wrong package name is used), so fix for script.bin will be delayed a bit.
I.e. install_kernel()
in boards.sh
tries to install linux-u-boot-next-sunxi_5.05_armhf.deb
, but real file name is linux-u-boot-next-cubietruck_5.05_armhf.deb
Since I want to load u-boot from rootfs rather than from sources directory, I need to fix this first.
Ok, I'm staying patient. Will do some hardware preparations in the meantime to be able to use FEL/NFS more efficiently. Did you had a look whether it's possible to FEL boot more than one device at the same time (especially when it's the same SoC? I'm currently away from the hardware and can't look myself but will later anyway)
Did you had a look whether it's possible to FEL boot more than one device at the same time (especially when it's the same SoC? I'm currently away from the hardware and can't look myself but will later anyway)
It depends on
Number 1 requires testing with more than one FEL capable device, which I can't do. And I don't see any command line parameters for specifying device by number or id anyway.
I tested 1) and since I might've overseen something tried to ask directly.
Regarding NFS mount options... the 1st method I thought about was how we setup cloned server environments (Solaris): There we create a snapshot of the FS in question, and create then a clone from it (at the fs layer). So in fact these are two independent filesystems that share most if not all blocks at the ZFS level. I will later have a look whether that's possible with btrfs in the meantime since that would be the way I prefer since you don't need to duplicate the rootfs image and add complexity only at a layer where it's not dangerous (compared to overlay fs or something like that).
I just realised that my new build host has a few more USB ports than expected. With 10 USB ports I can use one for the programmable power switch and 2 ports for each board (UART/FEL). Maybe I can figure out how to use the build host also to simulate an USB HID device. But one after the other :)
BTW: And I figured out how I can configure the small switch I will use in this setup to enable/disable ports so testing 1st boot situations with/without Ethernet can also be automated...
Here we go:
root@armbian:/usr/local/src/sunxi-tools# lsusb | grep Onda
Bus 008 Device 003: ID 1f3a:efe8 Onda (unverified) V972 tablet in flashing mode
Bus 003 Device 010: ID 1f3a:efe8 Onda (unverified) V972 tablet in flashing mode
root@armbian:/usr/local/src/sunxi-tools# ./fel -d 8:3 ver
AWUSBFEX soc=00001651(A20) 00000001 ver=0001 44 08 scratchpad=00007e00 00000000 00000000
root@armbian:/usr/local/src/sunxi-tools# ./fel -d 3:10 ver
AWUSBFEX soc=00001680(H3) 00000001 ver=0001 44 08 scratchpad=00007e00 00000000 00000000
root@armbian:/usr/local/src/sunxi-tools# ./fel -d 8:3 readl 0x01c23800
0x16516608
This is NextThing's fork with ssvb's patch to readout SID on top (looks now like this)
OK, right now I think the best way to use this is writing something like Igor's build-all.sh
script, which will take pairs of board names ($BOARD) and USB IDs and boot them one after another. Using multiple boards in parallel will require changes to build script anyway, and there are enough unmerged changes and ongoing issues already to test and fix.
I will give it a try in approx. one week. I still like the idea to boot a bunch of boards in parallel and to be able to automatically test through certain stuff (just ordered 5 of the UART dongles Igor recommended a while ago). And this could also be identical boards so $BOARD == $USB_ID won't work :)
Main showstopper now (at least for me): Most if not all boards stay powered on if you switch DC-IN off when they're connected with OTG to my build PC. Maybe a powered USB hub in between that can be power cycled through my Gembird power switch will help (FEL boot bandwidth is low enough).
I test it on my own and come back with ideas. BTW: in case you're interested in a few more sunxi devices I could send you Orange Pi One and one of the Pine64? I just have to wait for the Pine folks to send me their heatsink sample to finish the thermal testings...
And this could also be identical boards so $BOARD == $USB_ID won't work :)
By
pairs of board names ($BOARD) and USB IDs
I meant something like "cubietruck 1:2","orangepih3 3:4"
, and of course you would be able to use same board name with different IDs. I just don't see another way since it seems like we can't determine more than SoC type from SID info.
Main showstopper now (at least for me): Most if not all boards stay powered on if you switch DC-IN off when they're connected with OTG to my build PC. Maybe a powered USB hub in between that can be power cycled through my Gembird power switch will help (FEL boot bandwidth is low enough).
Boards with no onboard NAND and without SD card inserted should enter FEL mode automatically on power-on or after reboot, after that they are ready for upload. Boards with NAND flash could use special SD cards that triggers FEL mode on boot, but if you use connect serial adapters to them, you will probably run into "partial reset" problem and board may fail to detect SD card and boot from NAND instead.
Main showstopper for me (like I see now) - we have hardcoded path to rootfs, so booting different boards will require big changes to build script code. Also build script is designed for sequential building, so there may be problems running things in parallel, though it may be possible (and easier to setup) to run in containers (i.e. Wily and Xenial build hosts could use systemd-nspawn)
BTW: in case you're interested in a few more sunxi devices I could send you Orange Pi One and one of the Pine64?
Em, maybe?
I added hooks to FEL boot process: now you can create userpatches/fel-hooks.sh
and define there 2 functions: fel_post_prepare
and fel_pre_load
. All normal build variables like $BOARD, $BRANCH and so on can be used in these functions to define specific actions.
fel_post_prepare
is executed once after setting up u-boot script and NFS share, you can use it to add extra stuff to boot.scr (like gpio set
or setenv machid
) based on device name.
fel_pre_load
is executed before calling sunxi-fel
, you can use it to implement logic to select one of multiple connected boards; to pass additional arguments to sunxi-fel
you can use FEL_EXTRA_ARGS
variable.
Great! Thx a lot. Also for the other fixes, did quite a few FEL boots today that all worked flawlessly. I also hacked together a q&d hook and were able to FEL boot two different OPi PC directly after another.
Power switching the boards is now easy through fel_pre_load. I already thought about different ways to support more devices simultaneously but as you already said: Let's settle things a bit down first.
The ability to differentiate between devices based on USB bus/device ID will be included into official sunxi-tools with this patch :)
root@armbian:/# lsusb | sed -e 's/://' -e 's/0//g' | awk -F" " '/1f3a:efe8/ {print $2"\t"$4}' | while read; do set ${REPLY}; sunxi-fel --dev $1:$2 ver; done
AWUSBFEX soc=00001673(A83T) 00000001 ver=0001 44 08 scratchpad=00007e00 00000000 00000000
AWUSBFEX soc=00001651(A20) 00000001 ver=0001 44 08 scratchpad=00007e00 00000000 00000000
AWUSBFEX soc=00001680(H3) 00000001 ver=0001 44 08 scratchpad=00007e00 00000000 00000000
BTW: Another method that would also enhance the 'image personalisation install hook' I proposed in #231 would be to rely on FS features. Using btrfs for example we could use snapshots (they're writeable unlike with ZFS where you would have to make a clone from a snapshot first) to provide 3 identical rootfs for NFS boot without having to store this amount of data (only the writes from the clients will add up a little but the size of all 3 rootfs will stay close to the initial size of one).
And the same for image customisation as last step. Using btrfs/snapshots we would create one single image for all H3 board for example. Then create 4 snapshots, insert there script.bin/dtb, cpufreq and /etc/modules, let the images be closed and packed afterwards. No more need for 5 times the disk space since this is abstracted at the FS layer and the disk space needed for these 5 images will be the space needed for the first + 10k maybe.
But this is something I would only start when finally moving to Xenial as build host since being able to use a 4.x kernel is a requirement to use btrfs for me. And no need to hurry, just a thought that is worth a consideration when it's about building more than one image at a time (be it for distribution purposes or testing boards in parallel)
LOL, I remembered that in my 'new' build host (actually 6 years old) also two WD Raptor are installed. Justs created a RAID-0 (but FS metadata as RAID-1):
root@armbian:~# mkfs.btrfs -f -m raid1 -d raid0 /dev/sdb /dev/sdc
btrfs-progs v4.4
See http://btrfs.wiki.kernel.org for more information.
Label: (null)
UUID: cf4fa218-d190-446d-a838-e01aa35a3634
Node size: 16384
Sector size: 4096
Filesystem size: 279.47GiB
Block group profiles:
Data: RAID0 2.01GiB
Metadata: RAID1 1.01GiB
System: RAID1 12.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Number of devices: 2
Devices:
ID SIZE PATH
1 139.74GiB /dev/sdb
2 139.74GiB /dev/sdc
And mounted it with active compression (ZLIB on amd64), I get the usual write speeds (above 1100 MB/s since it's a hexacore i7) and okayish read speeds (250 MB/s). Next step is to make location of temporary image storage and NFS rootfs configurable. Using btrfs with compression should both increase performance when it's about image creation and also save space since the raw image only takes that much physical disk space as the ZIP version would need.
I'll play a bit around with this and might implement it as an optional tweak (something like _IMAGESTORAGE=/path/to/btrfs). And when I start testing real NFS boot I will rely on this btrfs/snapshot approach since anything else would just waste diskspace. With the snapshot feature providing a rootfs to 20 machines might result in needed diskspace of 1 rootfs + a few (hundred) MB written stuff on top.
Next step is to make location of temporary image storage and NFS rootfs configurable.
I think it's easier to mount --bind
or symlink output/images
to a location on btrfs volume - this way you don't need to modify build script at all.
It's not about output/images at all :)
I just introduced CACHEDIR=$DEST/cache in configuration.conf and exchanged every occurence of $DEST/cache with $CACHEDIR in the scripts since then you can easily set this in userpatches/lib.config but more importantly inside the scripts itself to adjust the behaviour (eg. switching between different btrfs subvolumes when doing the 'image finalisation' I talked about in #231).
I also plan to introduce CACHE_ON_BTRFS=yes|no (or auto detection -- see below) since when using btrfs it makes no sense to create .tgz archives inside cache/rootfs since you just store the rootfs in a subvolume (set to read-only) and when you create a new image you won't use pigz but instead create a writeable snapshot from the rootfs and install the necessary stuff therein (optionally do the image finalisation afterwards again on new snapshots). Needs less diskspace, consumes less cpu cycles and should speed things up (and starting with Xenial as build host using btrfs can be considered safe).
I thought defining $CACHEDIR now would be a good idea if we later start to adjust behaviour of the scripts depending on where $CACHEDIR is located. Any thoughts?
It's not about output/images at all :)
I understood your idea and I wanted to write that this will either require btrfs (which complicates initial build host setup or upgrade procedure) or will require significant changes to script code, and this complication helps mainly people who build images for multilpe boards.
Edit:
Needs less diskspace
Not exactly. If your definition of "old" build host means 8-core i7, then you can enable more efficient btrfs compression (zlib) and you won't notice any performance drop, while people with more crappy hardware may want to stick with lzo or no compression at all.
Sorry, I'm already in the advanced mode with 'image customisation' and multiple NFS/FEL boots in mind. Of course all this btrfs stuff has to be completely optional. But then you could do the following: Build one OS image for H3 and before calling _install_boardspecific function, you just create 4 snapshots for the fs $CACHEDIR is residing on, then adjust $CACHEDIR, walk through
for BOARD in orangepipc orangepiplus orangepi2 orangepione orangepilite ; do ...
install the few board specific support stuff in a few seconds and you're done (of course it's more complicated due to mount/umount but basically it's just that, isn't it?). And you don't need 5 times as much diskspace for 5 images created but only a little bit more even if you make 20 different images this way.
I mean I don't mind this as long as it's optional and doesn't complicate rest of build script much.
I will keep the btrfs stuff separate as an option for 'advanced builders' (but if we simply make build host == xenial
a requirement then it's pretty easy to use this stuff -- since it only affects the cache dir also host or filesystem changes aren't a problem since then simply the cache has gone).
I tested the hooks and added them to the documentation with #232 (and added also a small but working example for the hooks)
And BTW: I changed the fel boot behaviour to fel boot again when pressing any key != q
since it happened at least 5 times today that the wrong terminal window had the focus and I accidentally stopped FEL reboots by typing anything else than the needed in the wrong terminal window.
Just a small footnote (not needing any investigation but just to 'let you know'). When I tried out FEL/NFS boot with kernel 4.6-rc1 on sun8i only the first boot suceeded and all subsequent boots failed with 'nfs: server 192.168.83.120 not responding'.
I turned out this seems to be due to the Orange Pi PC using random MAC addresses on every reboot (new driver behaviour, might be fixed when we have H3 support in u-boot ready), therefore getting a new IP address through DHCP and then this happened. I assigned a static IP address through image customization and now everything works perfectly. :)
More a request for comments than a real issue. And it's about two issues that are related:
To speed up testing of images we could prevent burning the image to SD card completely -- at least on sunxi where FEL boot exists. In case we split the image creation process in a separate u-boot and a nfsroot capable rootfs then all that's needed for sunxi platforms where u-boot supports Ethernet (all except sun8i?) is an enhancement of the build process where the rootfs will automagically provided by NFS and u-boot pushed over USB to the connected sunxi device. Then instead of burning an SD card at the end of image creation you power cycle the board you want to test on before and as soon as image creation is ready sunxi-fel automagically transfers SPL/u-boot to the device which then loads the rest through NFS from the build host.
Different but related use case: Create images that work the same way but boot from SD card only containing u-boot and initiate from there loading kernel+rootfs through TFTP/NFS.
The efforts for both variants would be the same since based on my understanding you transfer through sunxi-fel what would be otherwise on a minimal (and pretty small in size) SD card for the 2nd variant.
Is anyone working on this already?