Should images be stored in Git LFS?

[tpimh]

Images are binary files, and they make up a significant part of the repository size. Each rootfs is 20Mb and each cpio is ~3Mb. Getting more architectures to build means also having more corresponding images. Given that we have 1Gb free LFS storage, maybe we can make use of it.

[nickdesaulniers]

So IIRC, @nathanchance created the images. Looking in driver.sh, seems like there's a mix of cpio and ext4 image use. If the cpio's are significantly smaller, can we drop the ext4 images? Do we need to append root= when using an initrd? (I think I was being a stick in the mud about keeping the ext4 images, but if I had a good reason to be, I no longer recall it)

To your point about binaries in git; these images are essentially never changing, or changing infrequently enough that they'd take up significant space, so it's not problematic yet, IMO. But I don't feel strongly about it; mostly I don't know anything about LFS storage. Can you point me to more info on how that would work?

[tpimh]

I don't know for sure about platform support, but pretty sure all should be able to use cpio images as initrd, so ext4 images can then be dropped.

In short, when using LFS selected files are not stored in repository, but instead are uploaded to a file server and a link to the file is stored in repository instead. Better explanation can be found in official docs.

[tpimh]

@nathanchance recently published updated images: https://github.com/ClangBuiltLinux/continuous-integration/commit/02fa71ea827d9127f729e228e848bc94d22cc9cf The repo is getting bigger with every update. Nathan, do you think we can drop ext4 images and only use cpio rootfs instead? Can we use Git LFS for cpio images in the future?

[nathanchance]

I suppose I am fine with dropping the ext4 images (although I am fairly certain we have caught bugs that way because we deal with an actual file system rather than just the ramdisk code).

The cpio image sizes are rather benign, do we actually need to use LFS? I'm just not exactly a huge fan of adding another binary to my workflow if I don't need to but if you all feel it's really beneficial, then sure.

3.0M    images/arm/rootfs.cpio
8.7M    images/ppc64le/rootfs.cpio
3.3M    images/arm64/rootfs.cpio
5.0M    images/ppc32/rootfs.cpio
3.5M    images/x86_64/rootfs.cpio

[tpimh]

Given that these images are not updated frequently and are pretty slim, it should be fine to proceed without LFS. How about gzipping them?

[nickdesaulniers]

Sure, why don't you see what makes them smallest; gzip, bzip, lz4 etc?

[tpimh]

Here is compression test of current rootfs.cpio for every arch:

	arm	arm64	mipsel	ppc32	ppc64	ppc64le	x86_64
no comp	3061248	3405824	3805696	5239296	9157632	9049600	3565568
bzip2 -9	1355957	1464769	1470082	1319283	2636797	2740250	1491278
gzip -9	1427459	1518362	1580781	1457225	2872146	2958789	1533389
lz4 -9	1676880	1794616	1856926	1695866	3325128	3393928	1788177
lzma -9 -e	1063855	1142821	1148776	1037374	2005783	2143598	1251130
lzop -9	1549072	1658336	1727513	1575432	3112088	3198560	1661583
xz -9 -e	1064064	1143040	1148996	1037572	2006136	2143972	1251372

LZMA is showing the best compression.

[nickdesaulniers]

The tradeoff in more efficient compression may be extended time to decompress, but if the tests aren't timing out, than I'm ok with crushing the images to be as small as possible.

[nathanchance]

I should write a hyperfine benchmark that does both compression and decompression times to see where we get the best all around results; I think that finding the right balance of decompression speed with space savings should be the goal because compression is only going to happen when we add new architectures or update the images (so not often) but decompression will happen all the time.

[nathanchance]

Okay, I got around to running some benchmarks :)

Results

Click for results

## arm (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 267.1 ± 3.2 | 261.9 | 271.8 | 2.58 ± 0.09 | 44% | | `gzip -9` | 428.4 ± 3.9 | 423.8 | 433.8 | 4.14 ± 0.14 | 46% | | `lz4 -9` | 103.6 ± 3.5 | 97.9 | 108.4 | 1.00 | 54% | | `lzma -9` | 1087.0 ± 6.0 | 1079.1 | 1104.1 | 10.49 ± 0.36 | 34% | | `lzop -9` | 611.4 ± 3.0 | 603.4 | 616.0 | 5.90 ± 0.20 | 50% | | `xz -9` | 1091.7 ± 6.3 | 1081.9 | 1106.9 | 10.54 ± 0.36 | 34% | | `zstd -19` | 673.0 ± 4.5 | 666.1 | 682.2 | 6.50 ± 0.22 | 40% | ## arm (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 155.0 ± 3.2 | 150.1 | 159.1 | 15.18 ± 2.82 | | `gzip -d` | 36.2 ± 2.9 | 30.9 | 40.6 | 3.55 ± 0.72 | | `lz4 -d` | 10.2 ± 1.9 | 5.5 | 12.9 | 1.00 | | `lzma -d` | 86.0 ± 2.8 | 81.0 | 89.6 | 8.43 ± 1.58 | | `lzop -d` | 19.9 ± 3.0 | 14.2 | 23.8 | 1.95 ± 0.47 | | `xz -d` | 88.5 ± 3.2 | 82.9 | 92.7 | 8.67 ± 1.63 | | `zstd -d` | 17.5 ± 2.9 | 11.8 | 22.2 | 1.71 ± 0.42 | ## arm64 (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 279.4 ± 3.2 | 273.5 | 284.5 | 2.57 ± 0.07 | 43% | | `gzip -9` | 551.7 ± 4.3 | 545.7 | 560.9 | 5.07 ± 0.14 | 44% | | `lz4 -9` | 108.7 ± 2.8 | 104.0 | 113.5 | 1.00 | 52% | | `lzma -9` | 1203.5 ± 4.0 | 1197.4 | 1216.2 | 11.07 ± 0.29 | 33% | | `lzop -9` | 818.5 ± 3.0 | 813.5 | 823.8 | 7.53 ± 0.20 | 48% | | `xz -9` | 1206.3 ± 3.7 | 1199.6 | 1213.1 | 11.09 ± 0.29 | 33% | | `zstd -19` | 730.1 ± 5.9 | 722.1 | 753.7 | 6.71 ± 0.18 | 39% | ## arm64 (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 161.2 ± 3.5 | 157.6 | 167.0 | 16.51 ± 4.04 | | `gzip -d` | 37.3 ± 3.3 | 33.0 | 41.4 | 3.82 ± 0.99 | | `lz4 -d` | 9.8 ± 2.4 | 5.9 | 13.7 | 1.00 | | `lzma -d` | 92.5 ± 3.1 | 86.6 | 95.6 | 9.47 ± 2.33 | | `lzop -d` | 21.0 ± 3.2 | 14.8 | 24.9 | 2.15 ± 0.62 | | `xz -d` | 95.5 ± 2.8 | 90.9 | 99.1 | 9.78 ± 2.40 | | `zstd -d` | 18.9 ± 3.1 | 13.3 | 23.0 | 1.93 ± 0.57 | ## mips (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 311.9 ± 3.4 | 307.3 | 316.4 | 2.26 ± 0.06 | 37% | | `gzip -9` | 712.1 ± 3.8 | 707.4 | 716.9 | 5.17 ± 0.13 | 41% | | `lz4 -9` | 137.8 ± 3.3 | 131.9 | 141.4 | 1.00 | 48% | | `lzma -9` | 1405.6 ± 5.2 | 1397.8 | 1421.2 | 10.20 ± 0.25 | 29% | | `lzop -9` | 878.2 ± 2.9 | 871.4 | 882.8 | 6.37 ± 0.15 | 45% | | `xz -9` | 1409.4 ± 4.5 | 1397.9 | 1420.5 | 10.23 ± 0.25 | 29% | | `zstd -19` | 889.9 ± 3.7 | 881.6 | 896.3 | 6.46 ± 0.16 | 35% | ## mips (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 174.8 ± 2.9 | 168.6 | 178.5 | 14.70 ± 3.29 | | `gzip -d` | 40.0 ± 3.2 | 35.5 | 45.0 | 3.36 ± 0.80 | | `lz4 -d` | 11.9 ± 2.7 | 5.9 | 15.1 | 1.00 | | `lzma -d` | 91.6 ± 3.9 | 85.6 | 96.3 | 7.70 ± 1.75 | | `lzop -d` | 22.6 ± 2.8 | 17.0 | 27.2 | 1.90 ± 0.49 | | `xz -d` | 97.3 ± 3.1 | 91.5 | 102.2 | 8.18 ± 1.84 | | `zstd -d` | 20.1 ± 3.0 | 13.9 | 25.0 | 1.69 ± 0.45 | ## mipsel (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 327.4 ± 3.4 | 323.0 | 332.8 | 2.41 ± 0.06 | 38% | | `gzip -9` | 508.9 ± 3.5 | 504.9 | 514.1 | 3.74 ± 0.08 | 41% | | `lz4 -9` | 136.0 ± 2.9 | 131.0 | 139.9 | 1.00 | 48% | | `lzma -9` | 1432.6 ± 4.7 | 1422.8 | 1439.9 | 10.54 ± 0.23 | 30% | | `lzop -9` | 809.0 ± 3.5 | 802.9 | 813.9 | 5.95 ± 0.13 | 45% | | `xz -9` | 1435.2 ± 4.5 | 1428.0 | 1444.3 | 10.55 ± 0.23 | 30% | | `zstd -19` | 886.6 ± 3.7 | 880.5 | 892.1 | 6.52 ± 0.14 | 35% | ## mipsel (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 179.8 ± 2.9 | 174.4 | 184.0 | 15.02 ± 2.81 | | `gzip -d` | 41.0 ± 3.5 | 35.9 | 45.0 | 3.43 ± 0.70 | | `lz4 -d` | 12.0 ± 2.2 | 6.6 | 16.5 | 1.00 | | `lzma -d` | 90.7 ± 3.1 | 84.7 | 94.9 | 7.57 ± 1.43 | | `lzop -d` | 22.1 ± 3.1 | 16.4 | 26.8 | 1.85 ± 0.43 | | `xz -d` | 95.7 ± 2.1 | 90.2 | 98.8 | 8.00 ± 1.50 | | `zstd -d` | 19.6 ± 3.0 | 13.8 | 23.4 | 1.64 ± 0.40 | ## ppc32 (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 450.5 ± 3.7 | 444.3 | 455.8 | 2.37 ± 0.04 | 25% | | `gzip -9` | 565.8 ± 3.7 | 561.8 | 570.8 | 2.98 ± 0.05 | 27% | | `lz4 -9` | 189.7 ± 3.2 | 183.4 | 193.3 | 1.00 | 32% | | `lzma -9` | 1118.2 ± 4.9 | 1109.9 | 1132.9 | 5.89 ± 0.10 | 19% | | `lzop -9` | 705.0 ± 3.7 | 697.7 | 710.7 | 3.72 ± 0.07 | 30% | | `xz -9` | 1122.8 ± 3.8 | 1117.2 | 1132.3 | 5.92 ± 0.10 | 19% | | `zstd -19` | 688.2 ± 3.3 | 684.9 | 694.5 | 3.63 ± 0.06 | 23% | ## ppc32 (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 155.4 ± 3.4 | 151.0 | 160.8 | 14.01 ± 3.77 | | `gzip -d` | 44.4 ± 3.6 | 40.2 | 50.0 | 4.00 ± 1.12 | | `lz4 -d` | 11.1 ± 3.0 | 6.5 | 16.1 | 1.00 | | `lzma -d` | 86.9 ± 3.2 | 82.4 | 91.5 | 7.83 ± 2.12 | | `lzop -d` | 24.3 ± 2.2 | 19.7 | 28.2 | 2.19 ± 0.62 | | `xz -d` | 93.1 ± 3.2 | 87.3 | 97.0 | 8.39 ± 2.27 | | `zstd -d` | 20.4 ± 2.6 | 14.0 | 24.1 | 1.84 ± 0.55 | ## ppc64 (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 618.8 ± 3.3 | 611.7 | 624.4 | 2.46 ± 0.03 | 28% | | `gzip -9` | 2176.2 ± 3.2 | 2171.0 | 2180.7 | 8.66 ± 0.10 | 31% | | `lz4 -9` | 251.4 ± 2.9 | 246.5 | 256.0 | 1.00 | 36% | | `lzma -9` | 2615.5 ± 7.8 | 2603.6 | 2631.8 | 10.40 ± 0.12 | 21% | | `lzop -9` | 2276.9 ± 4.2 | 2269.7 | 2283.7 | 9.06 ± 0.10 | 33% | | `xz -9` | 2623.6 ± 4.7 | 2614.4 | 2634.7 | 10.44 ± 0.12 | 21% | | `zstd -19` | 1725.3 ± 6.5 | 1715.4 | 1749.4 | 6.86 ± 0.08 | 25% | ## ppc64 (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 304.1 ± 3.0 | 299.8 | 308.4 | 20.86 ± 4.05 | | `gzip -d` | 77.1 ± 2.4 | 72.5 | 80.1 | 5.29 ± 1.04 | | `lz4 -d` | 14.6 ± 2.8 | 9.6 | 19.6 | 1.00 | | `lzma -d` | 161.2 ± 2.7 | 157.0 | 165.5 | 11.06 ± 2.15 | | `lzop -d` | 36.3 ± 3.0 | 32.2 | 41.2 | 2.49 ± 0.53 | | `xz -d` | 168.0 ± 3.1 | 163.8 | 173.1 | 11.52 ± 2.25 | | `zstd -d` | 30.1 ± 3.0 | 26.3 | 35.1 | 2.07 ± 0.45 | ## ppc64le (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 652.2 ± 4.2 | 646.1 | 659.8 | 2.58 ± 0.04 | 30% | | `gzip -9` | 1118.3 ± 3.7 | 1114.0 | 1123.6 | 4.42 ± 0.06 | 32% | | `lz4 -9` | 253.1 ± 3.3 | 248.5 | 257.8 | 1.00 | 37% | | `lzma -9` | 2602.8 ± 11.0 | 2584.7 | 2631.2 | 10.28 ± 0.14 | 23% | | `lzop -9` | 1722.7 ± 4.7 | 1717.5 | 1731.9 | 6.81 ± 0.09 | 35% | | `xz -9` | 2608.8 ± 7.1 | 2599.0 | 2623.7 | 10.31 ± 0.14 | 23% | | `zstd -19` | 1574.9 ± 12.7 | 1563.8 | 1619.3 | 6.22 ± 0.10 | 27% | ## ppc64le (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 319.3 ± 4.0 | 313.2 | 331.5 | 21.05 ± 3.52 | | `gzip -d` | 77.9 ± 2.2 | 72.0 | 80.7 | 5.14 ± 0.87 | | `lz4 -d` | 15.2 ± 2.5 | 9.6 | 18.8 | 1.00 | | `lzma -d` | 166.5 ± 2.4 | 161.0 | 170.9 | 10.98 ± 1.84 | | `lzop -d` | 36.7 ± 3.4 | 32.0 | 42.3 | 2.42 ± 0.46 | | `xz -d` | 175.4 ± 2.3 | 169.9 | 178.7 | 11.56 ± 1.93 | | `zstd -d` | 30.0 ± 3.2 | 26.6 | 36.1 | 1.97 ± 0.39 | ## x86_64 (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 268.9 ± 3.8 | 259.8 | 272.9 | 2.13 ± 0.06 | 41% | | `gzip -9` | 652.2 ± 3.4 | 647.4 | 656.9 | 5.17 ± 0.13 | 43% | | `lz4 -9` | 126.1 ± 3.2 | 120.5 | 129.5 | 1.00 | 50% | | `lzma -9` | 1196.0 ± 5.8 | 1189.3 | 1208.2 | 9.48 ± 0.24 | 35% | | `lzop -9` | 942.8 ± 3.9 | 937.5 | 950.6 | 7.48 ± 0.19 | 46% | | `xz -9` | 1201.0 ± 5.9 | 1193.6 | 1209.9 | 9.52 ± 0.25 | 35% | | `zstd -19` | 732.3 ± 2.8 | 727.5 | 737.5 | 5.81 ± 0.15 | 38% | ## x86_64 (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 165.4 ± 3.0 | 158.1 | 169.4 | 15.70 ± 3.72 | | `gzip -d` | 38.7 ± 3.3 | 33.3 | 43.0 | 3.67 ± 0.92 | | `lz4 -d` | 10.5 ± 2.5 | 5.5 | 14.1 | 1.00 | | `lzma -d` | 98.4 ± 3.6 | 93.5 | 104.0 | 9.34 ± 2.23 | | `lzop -d` | 20.3 ± 3.1 | 15.1 | 25.0 | 1.93 ± 0.54 | | `xz -d` | 100.7 ± 3.2 | 95.5 | 105.6 | 9.56 ± 2.28 | | `zstd -d` | 19.8 ± 2.2 | 13.9 | 23.3 | 1.88 ± 0.49 |

TL;DR: I believe we should use zstd because it has the second fastest decompression speed with the second most compression. The compression speed is the middle of the pack but better than lzma, which was worst in terms of decompression.

Now... the reason I did this now is that I am writing a set of regression/unit tests (basically what this repo does on a bit more of a grand scale) that will live in the tc-build repo but it needs these images for its boot tests. As a result, I propose that we split the images and buildroot folders off into their own repo, where we can compress them properly from the get go, then we just shallow clone that repo into this one in driver.sh initially, and I can use it for those tests in that repo. Thoughts? @tpimh @nickdesaulniers

Script

```bash #!/usr/bin/env bash BASE=$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd) CPIO_IMG=rootfs.cpio NINE_K_IMG=( -9 -k "${CPIO_IMG}" ) BZ2_COMP=( bzip2 "${NINE_K_IMG[@]}" ) GZ_COMP=( gzip "${NINE_K_IMG[@]}" ) LZ4_COMP=( lz4 -9 "${CPIO_IMG}" "${CPIO_IMG}".lz4 ) LZMA_COMP=( lzma "${NINE_K_IMG[@]}" ) LZOP_COMP=( lzop -9 "${CPIO_IMG}" ) XZ_COMP=( xz "${NINE_K_IMG[@]}" ) ZSTD_COMP=( zstd -19 "${CPIO_IMG}" ) # Create results folder RESULTS=${BASE}/results mkdir -p "${RESULTS}" # Benchmark compression for FOLDER in "${BASE}"/images/*; do COMP_RESULTS=${RESULTS}/${FOLDER##*/}-comp-results.md cd "${FOLDER}" && \ hyperfine --export-markdown ${COMP_RESULTS} \ --prepare "rm -vrf ${CPIO_IMG}.*" \ --warmup 1 \ --runs 25 \ "${BZ2_COMP[*]}" \ "${GZ_COMP[*]}" \ "${LZ4_COMP[*]}" \ "${LZMA_COMP[*]}" \ "${LZOP_COMP[*]}" \ "${XZ_COMP[*]}" \ "${ZSTD_COMP[*]}" && \ rm -vrf "${CPIO_IMG}".zst && \ "${BZ2_COMP[@]}" && \ "${GZ_COMP[@]}" && \ "${LZ4_COMP[@]}" && \ "${LZMA_COMP[@]}" && \ "${LZOP_COMP[@]}" && \ "${XZ_COMP[@]}" && \ "${ZSTD_COMP[@]}" && \ echo >> "${COMP_RESULTS}" LINE=3 for COMP_FILE in "${CPIO_IMG}".*; do PER_DIFF=$(echo "scale=2;$(stat --format=%s "${COMP_FILE}")/$(stat --format=%s "${COMP_FILE%.*}")*100" | bc | sed 's/\..*$/\%/') && \ sed -i "${LINE}s/$/ ${PER_DIFF} |/" "${COMP_RESULTS}" && \ LINE=$(( LINE + 1 )) done hyperfine --export-markdown ${RESULTS}/${FOLDER##*/}-decomp-results.md \ --prepare "rm -vrf ${CPIO_IMG}" \ --warmup 1 \ --runs 25 \ "bzip2 -d -k ${CPIO_IMG}.bz2" \ "gzip -d -k ${CPIO_IMG}.gz" \ "lz4 -d ${CPIO_IMG}.lz4" \ "lzma -d -k ${CPIO_IMG}.lzma" \ "lzop -d ${CPIO_IMG}.lzo" \ "xz -d -k ${CPIO_IMG}.xz" \ "zstd -d ${CPIO_IMG}.zst" done # Format files for COMP_RESULT in "${RESULTS}"/*-comp-results*; do sed -i -e 's/Relative |/Relative speed | Relative size |/' -e '2s/$/---:|/' -e 's/9 .*r.*`/9`/' -e 's/9 .* -k.*`/9`/' "${COMP_RESULT}" done for DECOMP_RESULT in "${RESULTS}"/*-decomp-results*; do sed -i 's/ -d.*` / -d` /' "${DECOMP_RESULT}" done ```

[tpimh]

I don't think we really should care about the compression speed as it's done so rarely the expenses can be neglected.

Ideally we can move buildroot and images directories to a separate repository (possibly, even preserving their git history), then set the images up to be built on travis if any of the configs is changed like we do with docker images.

[nickdesaulniers]

No preference on choice of compression. As long as we don't have to modify defconfigs to decompress, I'm happy. Just pick one; not multiple, if possible.

As a result, I propose that we split the images and buildroot folders off into their own repo

SGTM. Two repos, or one repo? I'm fine with either, but it's unclear to me as stated.

possibly, even preserving their git history

I'm indifferent on this point.

I don't think we really should care about the compression speed as it's done so rarely the expenses can be neglected.

That's a fair point.

Orthogonally, I've been finding it frustrating to:

1. jump on a new machine without shell history, and try to recreate our QEMU commands from driver.sh. Even sending commands to newbies is a PITA.
2. boot more than just our scripts that immediately power off a machine.

I think it would be helpful for us to provide:

1. userspace images that just powers off the machine
2. userspace images that just has a basic shell/toybox
3. full blown debian image
4. prebuilt clang built kernel images
5. a command line utility that uses some combination of the above

I don't know what's a good way to organize these, since our CI probably doesn't care about 2-5, but am generally a fan of git submodules (as long as the readme clearly states the correct way to recursively clone, because I always forget to do that). Some can even likely just be fetched lazily (ie. debian images).

In particular, I currently find it difficult to do more userspace testing of clang built kernels, and I think the above would go a long way towards helping. And that doesn't even touch upon packaging kernel modules from a clang built kernel build.

[nathanchance]

SGTM. Two repos, or one repo? I'm fine with either, but it's unclear to me as stated.

I am planning one repo:

https://github.com/ClangBuiltLinux/boot-utils

https://github.com/nathanchance/boot-utils/commit/1740f54509628e5e2dd5e2c5a01511097b96df73

I will PR when ready for review.

possibly, even preserving their git history

I'm indifferent on this point.

I cannot preserve buildroot's history with git filter-branch (link) because of https://github.com/ClangBuiltLinux/continuous-integration/commit/b2848d88fddcaa84e6d0b1473df6f06cf0f6992b and I'd rather not use another tool. We can just link to the history on GitHub like so:

https://github.com/ClangBuiltLinux/continuous-integration/commits/f49b9a3e4a9a1cfa1164e5542d6f62668759218c/buildroot

I do not think preserving images is worthwhile.

I don't think we really should care about the compression speed as it's done so rarely the expenses can be neglected.

Agreed. The balance of decompression speed with regards to ratio is more important.

Orthogonally, I've been finding it frustrating to:

1. jump on a new machine without shell history, and try to recreate our QEMU commands from driver.sh. Even sending commands to newbies is a PITA.
2. boot more than just our scripts that immediately power off a machine.

Agreed. Maybe we move the QEMU commands into a separate script within the rootfs repo above (and rename it boot-testing?) so that all we have to do is supply a kernel image and it just boots it for us?

Something like:

./boot-qemu.sh <arch>

and maybe an argument like --shell for when we want to be dumped into a shell?

EDIT: Renamed the rootfs-images repo to boot-utils as this makes life a lot easier.

I think it would be helpful for us to provide:

1. userspace images that just powers off the machine
2. userspace images that just has a basic shell/toybox

Our images can do both of these. The second one is just done by passing rdinit=/bin/sh on the command line.

$ qemu-system-x86_64 -append 'console=ttyS0 rdinit=/bin/sh' -cpu host -d unimp,guest_errors -initrd ../../cbl/git/ci-master/images/x86_64/rootfs.cpio -enable-kvm -kernel out/x86_64/arch/x86/boot/bzImage -m 512m -display none -serial mon:stdio
...
/ # mount proc /proc -t proc
/ # cat /proc/version
Linux version 5.6.0-rc7+ (nathan@ubuntu-m2-xlarge-x86) (ClangBuiltLinux clang version 11.0.0 (git://github.com/llvm/llvm-project d264f02c6f502960e2bcdd332f250efc702d09f2)) #1 SMP Thu Mar 26 15:16:14 MST 2020
/ # ls
bin      init     linuxrc  opt      run      tmp
dev      lib      media    proc     sbin     usr
etc      lib64    mnt      root     sys      var
/ #

3. full blown debian image

I do not know that we have the bandwidth for this.

4. prebuilt clang built kernel images
5. a command line utility that uses some combination of the above

See above.

I don't know what's a good way to organize these, since our CI probably doesn't care about 2-5, but am generally a fan of git submodules (as long as the readme clearly states the correct way to recursively clone, because I always forget to do that). Some can even likely just be fetched lazily (ie. debian images).

Submodules do not play well with Travis unfortunately but cloning the repos and managing them through driver.sh is not too painful.

In particular, I currently find it difficult to do more userspace testing of clang built kernels, and I think the above would go a long way towards helping. And that doesn't even touch upon packaging kernel modules from a clang built kernel build.

I'll whip something up.

[nathanchance]

https://github.com/ClangBuiltLinux/boot-utils/pull/1 https://github.com/ClangBuiltLinux/continuous-integration/pull/244 https://github.com/ClangBuiltLinux/dockerimage/pull/43

[tpimh]

When was zstd decompression support added to the kernel?

[nathanchance]

It hasn’t been yet: https://lore.kernel.org/lkml/20200325195849.407900-1-nickrterrell@gmail.com/

[tpimh]

So I guess lzma is a better option: faster decompression and support for older kernels.

[nathanchance]

We can still use zstd and just decompress the images on the fly before using them. It only takes a maximum of 26ms in my testa above.

That is what I am doing here: https://github.com/nathanchance/boot-utils/blob/updates/boot-qemu.sh#L82-L83

Please review the PR if you have any other concerns, I'll leave it open until Monday evening then merge it: https://github.com/ClangBuiltLinux/boot-utils/pull/1