support for NAND and block-based devices

[dangowrt]

currently only NOR-based devices with squashfs-sysupgrade images are supported. Support also devices which are block-device or NAND-flash based by picking up the required files generated by the ImageBuilder.

useful for sysupgrade:

• most NOR based devices: *-${FSTYPE}-sysupgrade.bin
• Broadcom NOR devices: *-${FSTYPE}.trx, *-${FSTYPE}.chk, *-${FSTYPE}-sysupgrade.bin, *-${FSTYPE}.bin
• NAND based devices: *-${FSTYPE}-sysupgrade.tar
• MMC based devices: *-${FSTYPE}-sdcard.img.gz
• x86 MBR: *-combined-${FSTYPE}*
• x86 GPT: *-uefi-gpt-${FSTYPE}*

Hence, to pick the right image for sysupgrade, we need to know the rootfs-type (squashfs vs. jffs2/ubifs/ext4/f2fs) as well as the type of the rootdisk's partition table on x86-based devices (MBR vs. GPT/UEFI).

As there are images files generated for other purposes (factory, ...) it makes sense to serve them as well for use with jefe.tk. To achieve that, I suggest to simply match all filenames containing the EXTRA_IMAGE_NAME substring supplied to IB.

[aparcar]

@dangowrt so how will the client (e.g. luci-app-attendedsysupgrade) know what file is the actual upgrade? possible approach is to implement update-request/ and build-request/, while update only return url of sysupgrade, filesize and checksum the build-request contains a array of all created files. the update-server (maybe from now on image-server?) has to determine itself what the sysupgrade file is

[dangowrt]

All files (with their sha256sums signed) should be served to allow use-cases other than attended-sysupgrade. And yes, the build server should pick the squashfs variant matching the sysupgrade file scheme of that target. The knowledge which image file that is could be a list compared top-down for the first matching glob by the image server to pick which image to sign and suggest to the client:

*-squashfs-sysupgrade.bin
*-squashfs-sysupgrade.tar
*-squashfs.trx
*-squashfs.chk
*-squashfs.bin
*-squashfs-sdcard.img.gz
*-combined-squashfs*

x86 is going to be re-structured at some point hopefully, so the more logical thing would be to have different boards/devices like generic-uefi, generic-mbr, generic-iso, vmware-mbr (which could have e.g. different kernel parameters or bootloader options and returning a vmdk image), ... Until then, I suggest to simply only offer the *-combined-squashfs* image as that's what works for most people.

[aparcar]

Great, the sysupgrade file is now detected based in the list above. currently the server unifies the names of sysupgrade files. however, having various files per build needs a fast way to return them. Storing them in database or using a (python) glob could be an approach

[aparcar]

All builds store now all files, an example here: https://betaupdate.libremesh.org/static/lede/17.01.3/ar71xx/generic/tl-wdr4300-v1/ I'll create another API call (and pretty up nginx) to make these files accessible

[aparcar]

@dangowrt could you please provide the same list as above for factory images?

[dangowrt]

hm, it's not that trivial, because it depends on the factory flashing method (web-ui, tftp-server, tftp-client, ...) as well as the flash layout of the stock firmware (hence, sometimes multiple or intermediate images are needed, especially for NAND-based devices). the safest thing is to server all generated files as the device-specific documentation may refer to a subset of them.

[aparcar]

instead of storing the sysupgrade filename in the database a file link from sysupgrade.bin to <complicated_sysupgrade_name_here>.<something> could be created. Clients receive a url to all created files. The sysupgrade client simply requests url/sysupgrade.bin. @dangowrt

[aparcar]

https://github.com/aparcar/gsoc17-attended-sysupgrade/issues/56

[aparcar]

fixed with 767c192188db307c20843b0156ada6806160b44e 56f3daf183a41a6aba678923cd2ad0e627822db7 and cfff9735b4cc7131b8f7d0b0f5cf6bdb2cd1e424

images are no longer renamed. This still can lead to problem when the imagesbuilder names images in an odd way, like leaving the subtarget away, however mho this should be unified instead of creating an endless number of cases with different naming styles.

@dangowrt please close the issue when it fits you!

[aparcar]

should be all done