Open tpimh opened 5 years ago
So IIRC, @nathanchance created the images. Looking in driver.sh, seems like there's a mix of cpio and ext4 image use. If the cpio's are significantly smaller, can we drop the ext4 images? Do we need to append root=
when using an initrd
? (I think I was being a stick in the mud about keeping the ext4 images, but if I had a good reason to be, I no longer recall it)
To your point about binaries in git; these images are essentially never changing, or changing infrequently enough that they'd take up significant space, so it's not problematic yet, IMO. But I don't feel strongly about it; mostly I don't know anything about LFS storage
. Can you point me to more info on how that would work?
I don't know for sure about platform support, but pretty sure all should be able to use cpio images as initrd, so ext4 images can then be dropped.
In short, when using LFS selected files are not stored in repository, but instead are uploaded to a file server and a link to the file is stored in repository instead. Better explanation can be found in official docs.
@nathanchance recently published updated images: https://github.com/ClangBuiltLinux/continuous-integration/commit/02fa71ea827d9127f729e228e848bc94d22cc9cf The repo is getting bigger with every update. Nathan, do you think we can drop ext4 images and only use cpio rootfs instead? Can we use Git LFS for cpio images in the future?
I suppose I am fine with dropping the ext4 images (although I am fairly certain we have caught bugs that way because we deal with an actual file system rather than just the ramdisk code).
The cpio image sizes are rather benign, do we actually need to use LFS? I'm just not exactly a huge fan of adding another binary to my workflow if I don't need to but if you all feel it's really beneficial, then sure.
3.0M images/arm/rootfs.cpio
8.7M images/ppc64le/rootfs.cpio
3.3M images/arm64/rootfs.cpio
5.0M images/ppc32/rootfs.cpio
3.5M images/x86_64/rootfs.cpio
Given that these images are not updated frequently and are pretty slim, it should be fine to proceed without LFS. How about gzipping them?
Sure, why don't you see what makes them smallest; gzip, bzip, lz4 etc?
Here is compression test of current rootfs.cpio for every arch:
arm | arm64 | mipsel | ppc32 | ppc64 | ppc64le | x86_64 | |
---|---|---|---|---|---|---|---|
no comp | 3061248 | 3405824 | 3805696 | 5239296 | 9157632 | 9049600 | 3565568 |
bzip2 -9 |
1355957 | 1464769 | 1470082 | 1319283 | 2636797 | 2740250 | 1491278 |
gzip -9 |
1427459 | 1518362 | 1580781 | 1457225 | 2872146 | 2958789 | 1533389 |
lz4 -9 |
1676880 | 1794616 | 1856926 | 1695866 | 3325128 | 3393928 | 1788177 |
lzma -9 -e |
1063855 | 1142821 | 1148776 | 1037374 | 2005783 | 2143598 | 1251130 |
lzop -9 |
1549072 | 1658336 | 1727513 | 1575432 | 3112088 | 3198560 | 1661583 |
xz -9 -e |
1064064 | 1143040 | 1148996 | 1037572 | 2006136 | 2143972 | 1251372 |
LZMA is showing the best compression.
The tradeoff in more efficient compression may be extended time to decompress, but if the tests aren't timing out, than I'm ok with crushing the images to be as small as possible.
I should write a hyperfine
benchmark that does both compression and decompression times to see where we get the best all around results; I think that finding the right balance of decompression speed with space savings should be the goal because compression is only going to happen when we add new architectures or update the images (so not often) but decompression will happen all the time.
Okay, I got around to running some benchmarks :)
## arm (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 267.1 ± 3.2 | 261.9 | 271.8 | 2.58 ± 0.09 | 44% | | `gzip -9` | 428.4 ± 3.9 | 423.8 | 433.8 | 4.14 ± 0.14 | 46% | | `lz4 -9` | 103.6 ± 3.5 | 97.9 | 108.4 | 1.00 | 54% | | `lzma -9` | 1087.0 ± 6.0 | 1079.1 | 1104.1 | 10.49 ± 0.36 | 34% | | `lzop -9` | 611.4 ± 3.0 | 603.4 | 616.0 | 5.90 ± 0.20 | 50% | | `xz -9` | 1091.7 ± 6.3 | 1081.9 | 1106.9 | 10.54 ± 0.36 | 34% | | `zstd -19` | 673.0 ± 4.5 | 666.1 | 682.2 | 6.50 ± 0.22 | 40% | ## arm (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 155.0 ± 3.2 | 150.1 | 159.1 | 15.18 ± 2.82 | | `gzip -d` | 36.2 ± 2.9 | 30.9 | 40.6 | 3.55 ± 0.72 | | `lz4 -d` | 10.2 ± 1.9 | 5.5 | 12.9 | 1.00 | | `lzma -d` | 86.0 ± 2.8 | 81.0 | 89.6 | 8.43 ± 1.58 | | `lzop -d` | 19.9 ± 3.0 | 14.2 | 23.8 | 1.95 ± 0.47 | | `xz -d` | 88.5 ± 3.2 | 82.9 | 92.7 | 8.67 ± 1.63 | | `zstd -d` | 17.5 ± 2.9 | 11.8 | 22.2 | 1.71 ± 0.42 | ## arm64 (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 279.4 ± 3.2 | 273.5 | 284.5 | 2.57 ± 0.07 | 43% | | `gzip -9` | 551.7 ± 4.3 | 545.7 | 560.9 | 5.07 ± 0.14 | 44% | | `lz4 -9` | 108.7 ± 2.8 | 104.0 | 113.5 | 1.00 | 52% | | `lzma -9` | 1203.5 ± 4.0 | 1197.4 | 1216.2 | 11.07 ± 0.29 | 33% | | `lzop -9` | 818.5 ± 3.0 | 813.5 | 823.8 | 7.53 ± 0.20 | 48% | | `xz -9` | 1206.3 ± 3.7 | 1199.6 | 1213.1 | 11.09 ± 0.29 | 33% | | `zstd -19` | 730.1 ± 5.9 | 722.1 | 753.7 | 6.71 ± 0.18 | 39% | ## arm64 (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 161.2 ± 3.5 | 157.6 | 167.0 | 16.51 ± 4.04 | | `gzip -d` | 37.3 ± 3.3 | 33.0 | 41.4 | 3.82 ± 0.99 | | `lz4 -d` | 9.8 ± 2.4 | 5.9 | 13.7 | 1.00 | | `lzma -d` | 92.5 ± 3.1 | 86.6 | 95.6 | 9.47 ± 2.33 | | `lzop -d` | 21.0 ± 3.2 | 14.8 | 24.9 | 2.15 ± 0.62 | | `xz -d` | 95.5 ± 2.8 | 90.9 | 99.1 | 9.78 ± 2.40 | | `zstd -d` | 18.9 ± 3.1 | 13.3 | 23.0 | 1.93 ± 0.57 | ## mips (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 311.9 ± 3.4 | 307.3 | 316.4 | 2.26 ± 0.06 | 37% | | `gzip -9` | 712.1 ± 3.8 | 707.4 | 716.9 | 5.17 ± 0.13 | 41% | | `lz4 -9` | 137.8 ± 3.3 | 131.9 | 141.4 | 1.00 | 48% | | `lzma -9` | 1405.6 ± 5.2 | 1397.8 | 1421.2 | 10.20 ± 0.25 | 29% | | `lzop -9` | 878.2 ± 2.9 | 871.4 | 882.8 | 6.37 ± 0.15 | 45% | | `xz -9` | 1409.4 ± 4.5 | 1397.9 | 1420.5 | 10.23 ± 0.25 | 29% | | `zstd -19` | 889.9 ± 3.7 | 881.6 | 896.3 | 6.46 ± 0.16 | 35% | ## mips (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 174.8 ± 2.9 | 168.6 | 178.5 | 14.70 ± 3.29 | | `gzip -d` | 40.0 ± 3.2 | 35.5 | 45.0 | 3.36 ± 0.80 | | `lz4 -d` | 11.9 ± 2.7 | 5.9 | 15.1 | 1.00 | | `lzma -d` | 91.6 ± 3.9 | 85.6 | 96.3 | 7.70 ± 1.75 | | `lzop -d` | 22.6 ± 2.8 | 17.0 | 27.2 | 1.90 ± 0.49 | | `xz -d` | 97.3 ± 3.1 | 91.5 | 102.2 | 8.18 ± 1.84 | | `zstd -d` | 20.1 ± 3.0 | 13.9 | 25.0 | 1.69 ± 0.45 | ## mipsel (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 327.4 ± 3.4 | 323.0 | 332.8 | 2.41 ± 0.06 | 38% | | `gzip -9` | 508.9 ± 3.5 | 504.9 | 514.1 | 3.74 ± 0.08 | 41% | | `lz4 -9` | 136.0 ± 2.9 | 131.0 | 139.9 | 1.00 | 48% | | `lzma -9` | 1432.6 ± 4.7 | 1422.8 | 1439.9 | 10.54 ± 0.23 | 30% | | `lzop -9` | 809.0 ± 3.5 | 802.9 | 813.9 | 5.95 ± 0.13 | 45% | | `xz -9` | 1435.2 ± 4.5 | 1428.0 | 1444.3 | 10.55 ± 0.23 | 30% | | `zstd -19` | 886.6 ± 3.7 | 880.5 | 892.1 | 6.52 ± 0.14 | 35% | ## mipsel (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 179.8 ± 2.9 | 174.4 | 184.0 | 15.02 ± 2.81 | | `gzip -d` | 41.0 ± 3.5 | 35.9 | 45.0 | 3.43 ± 0.70 | | `lz4 -d` | 12.0 ± 2.2 | 6.6 | 16.5 | 1.00 | | `lzma -d` | 90.7 ± 3.1 | 84.7 | 94.9 | 7.57 ± 1.43 | | `lzop -d` | 22.1 ± 3.1 | 16.4 | 26.8 | 1.85 ± 0.43 | | `xz -d` | 95.7 ± 2.1 | 90.2 | 98.8 | 8.00 ± 1.50 | | `zstd -d` | 19.6 ± 3.0 | 13.8 | 23.4 | 1.64 ± 0.40 | ## ppc32 (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 450.5 ± 3.7 | 444.3 | 455.8 | 2.37 ± 0.04 | 25% | | `gzip -9` | 565.8 ± 3.7 | 561.8 | 570.8 | 2.98 ± 0.05 | 27% | | `lz4 -9` | 189.7 ± 3.2 | 183.4 | 193.3 | 1.00 | 32% | | `lzma -9` | 1118.2 ± 4.9 | 1109.9 | 1132.9 | 5.89 ± 0.10 | 19% | | `lzop -9` | 705.0 ± 3.7 | 697.7 | 710.7 | 3.72 ± 0.07 | 30% | | `xz -9` | 1122.8 ± 3.8 | 1117.2 | 1132.3 | 5.92 ± 0.10 | 19% | | `zstd -19` | 688.2 ± 3.3 | 684.9 | 694.5 | 3.63 ± 0.06 | 23% | ## ppc32 (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 155.4 ± 3.4 | 151.0 | 160.8 | 14.01 ± 3.77 | | `gzip -d` | 44.4 ± 3.6 | 40.2 | 50.0 | 4.00 ± 1.12 | | `lz4 -d` | 11.1 ± 3.0 | 6.5 | 16.1 | 1.00 | | `lzma -d` | 86.9 ± 3.2 | 82.4 | 91.5 | 7.83 ± 2.12 | | `lzop -d` | 24.3 ± 2.2 | 19.7 | 28.2 | 2.19 ± 0.62 | | `xz -d` | 93.1 ± 3.2 | 87.3 | 97.0 | 8.39 ± 2.27 | | `zstd -d` | 20.4 ± 2.6 | 14.0 | 24.1 | 1.84 ± 0.55 | ## ppc64 (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 618.8 ± 3.3 | 611.7 | 624.4 | 2.46 ± 0.03 | 28% | | `gzip -9` | 2176.2 ± 3.2 | 2171.0 | 2180.7 | 8.66 ± 0.10 | 31% | | `lz4 -9` | 251.4 ± 2.9 | 246.5 | 256.0 | 1.00 | 36% | | `lzma -9` | 2615.5 ± 7.8 | 2603.6 | 2631.8 | 10.40 ± 0.12 | 21% | | `lzop -9` | 2276.9 ± 4.2 | 2269.7 | 2283.7 | 9.06 ± 0.10 | 33% | | `xz -9` | 2623.6 ± 4.7 | 2614.4 | 2634.7 | 10.44 ± 0.12 | 21% | | `zstd -19` | 1725.3 ± 6.5 | 1715.4 | 1749.4 | 6.86 ± 0.08 | 25% | ## ppc64 (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 304.1 ± 3.0 | 299.8 | 308.4 | 20.86 ± 4.05 | | `gzip -d` | 77.1 ± 2.4 | 72.5 | 80.1 | 5.29 ± 1.04 | | `lz4 -d` | 14.6 ± 2.8 | 9.6 | 19.6 | 1.00 | | `lzma -d` | 161.2 ± 2.7 | 157.0 | 165.5 | 11.06 ± 2.15 | | `lzop -d` | 36.3 ± 3.0 | 32.2 | 41.2 | 2.49 ± 0.53 | | `xz -d` | 168.0 ± 3.1 | 163.8 | 173.1 | 11.52 ± 2.25 | | `zstd -d` | 30.1 ± 3.0 | 26.3 | 35.1 | 2.07 ± 0.45 | ## ppc64le (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 652.2 ± 4.2 | 646.1 | 659.8 | 2.58 ± 0.04 | 30% | | `gzip -9` | 1118.3 ± 3.7 | 1114.0 | 1123.6 | 4.42 ± 0.06 | 32% | | `lz4 -9` | 253.1 ± 3.3 | 248.5 | 257.8 | 1.00 | 37% | | `lzma -9` | 2602.8 ± 11.0 | 2584.7 | 2631.2 | 10.28 ± 0.14 | 23% | | `lzop -9` | 1722.7 ± 4.7 | 1717.5 | 1731.9 | 6.81 ± 0.09 | 35% | | `xz -9` | 2608.8 ± 7.1 | 2599.0 | 2623.7 | 10.31 ± 0.14 | 23% | | `zstd -19` | 1574.9 ± 12.7 | 1563.8 | 1619.3 | 6.22 ± 0.10 | 27% | ## ppc64le (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 319.3 ± 4.0 | 313.2 | 331.5 | 21.05 ± 3.52 | | `gzip -d` | 77.9 ± 2.2 | 72.0 | 80.7 | 5.14 ± 0.87 | | `lz4 -d` | 15.2 ± 2.5 | 9.6 | 18.8 | 1.00 | | `lzma -d` | 166.5 ± 2.4 | 161.0 | 170.9 | 10.98 ± 1.84 | | `lzop -d` | 36.7 ± 3.4 | 32.0 | 42.3 | 2.42 ± 0.46 | | `xz -d` | 175.4 ± 2.3 | 169.9 | 178.7 | 11.56 ± 1.93 | | `zstd -d` | 30.0 ± 3.2 | 26.6 | 36.1 | 1.97 ± 0.39 | ## x86_64 (compression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative speed | Relative size | |:---|---:|---:|---:|---:|---:| | `bzip2 -9` | 268.9 ± 3.8 | 259.8 | 272.9 | 2.13 ± 0.06 | 41% | | `gzip -9` | 652.2 ± 3.4 | 647.4 | 656.9 | 5.17 ± 0.13 | 43% | | `lz4 -9` | 126.1 ± 3.2 | 120.5 | 129.5 | 1.00 | 50% | | `lzma -9` | 1196.0 ± 5.8 | 1189.3 | 1208.2 | 9.48 ± 0.24 | 35% | | `lzop -9` | 942.8 ± 3.9 | 937.5 | 950.6 | 7.48 ± 0.19 | 46% | | `xz -9` | 1201.0 ± 5.9 | 1193.6 | 1209.9 | 9.52 ± 0.25 | 35% | | `zstd -19` | 732.3 ± 2.8 | 727.5 | 737.5 | 5.81 ± 0.15 | 38% | ## x86_64 (decompression) | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `bzip2 -d` | 165.4 ± 3.0 | 158.1 | 169.4 | 15.70 ± 3.72 | | `gzip -d` | 38.7 ± 3.3 | 33.3 | 43.0 | 3.67 ± 0.92 | | `lz4 -d` | 10.5 ± 2.5 | 5.5 | 14.1 | 1.00 | | `lzma -d` | 98.4 ± 3.6 | 93.5 | 104.0 | 9.34 ± 2.23 | | `lzop -d` | 20.3 ± 3.1 | 15.1 | 25.0 | 1.93 ± 0.54 | | `xz -d` | 100.7 ± 3.2 | 95.5 | 105.6 | 9.56 ± 2.28 | | `zstd -d` | 19.8 ± 2.2 | 13.9 | 23.3 | 1.88 ± 0.49 |
TL;DR: I believe we should use zstd
because it has the second fastest decompression speed with the second most compression. The compression speed is the middle of the pack but better than lzma
, which was worst in terms of decompression.
Now... the reason I did this now is that I am writing a set of regression/unit tests (basically what this repo does on a bit more of a grand scale) that will live in the tc-build
repo but it needs these images for its boot tests. As a result, I propose that we split the images
and buildroot
folders off into their own repo, where we can compress them properly from the get go, then we just shallow clone that repo into this one in driver.sh
initially, and I can use it for those tests in that repo. Thoughts? @tpimh @nickdesaulniers
```bash #!/usr/bin/env bash BASE=$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd) CPIO_IMG=rootfs.cpio NINE_K_IMG=( -9 -k "${CPIO_IMG}" ) BZ2_COMP=( bzip2 "${NINE_K_IMG[@]}" ) GZ_COMP=( gzip "${NINE_K_IMG[@]}" ) LZ4_COMP=( lz4 -9 "${CPIO_IMG}" "${CPIO_IMG}".lz4 ) LZMA_COMP=( lzma "${NINE_K_IMG[@]}" ) LZOP_COMP=( lzop -9 "${CPIO_IMG}" ) XZ_COMP=( xz "${NINE_K_IMG[@]}" ) ZSTD_COMP=( zstd -19 "${CPIO_IMG}" ) # Create results folder RESULTS=${BASE}/results mkdir -p "${RESULTS}" # Benchmark compression for FOLDER in "${BASE}"/images/*; do COMP_RESULTS=${RESULTS}/${FOLDER##*/}-comp-results.md cd "${FOLDER}" && \ hyperfine --export-markdown ${COMP_RESULTS} \ --prepare "rm -vrf ${CPIO_IMG}.*" \ --warmup 1 \ --runs 25 \ "${BZ2_COMP[*]}" \ "${GZ_COMP[*]}" \ "${LZ4_COMP[*]}" \ "${LZMA_COMP[*]}" \ "${LZOP_COMP[*]}" \ "${XZ_COMP[*]}" \ "${ZSTD_COMP[*]}" && \ rm -vrf "${CPIO_IMG}".zst && \ "${BZ2_COMP[@]}" && \ "${GZ_COMP[@]}" && \ "${LZ4_COMP[@]}" && \ "${LZMA_COMP[@]}" && \ "${LZOP_COMP[@]}" && \ "${XZ_COMP[@]}" && \ "${ZSTD_COMP[@]}" && \ echo >> "${COMP_RESULTS}" LINE=3 for COMP_FILE in "${CPIO_IMG}".*; do PER_DIFF=$(echo "scale=2;$(stat --format=%s "${COMP_FILE}")/$(stat --format=%s "${COMP_FILE%.*}")*100" | bc | sed 's/\..*$/\%/') && \ sed -i "${LINE}s/$/ ${PER_DIFF} |/" "${COMP_RESULTS}" && \ LINE=$(( LINE + 1 )) done hyperfine --export-markdown ${RESULTS}/${FOLDER##*/}-decomp-results.md \ --prepare "rm -vrf ${CPIO_IMG}" \ --warmup 1 \ --runs 25 \ "bzip2 -d -k ${CPIO_IMG}.bz2" \ "gzip -d -k ${CPIO_IMG}.gz" \ "lz4 -d ${CPIO_IMG}.lz4" \ "lzma -d -k ${CPIO_IMG}.lzma" \ "lzop -d ${CPIO_IMG}.lzo" \ "xz -d -k ${CPIO_IMG}.xz" \ "zstd -d ${CPIO_IMG}.zst" done # Format files for COMP_RESULT in "${RESULTS}"/*-comp-results*; do sed -i -e 's/Relative |/Relative speed | Relative size |/' -e '2s/$/---:|/' -e 's/9 .*r.*`/9`/' -e 's/9 .* -k.*`/9`/' "${COMP_RESULT}" done for DECOMP_RESULT in "${RESULTS}"/*-decomp-results*; do sed -i 's/ -d.*` / -d` /' "${DECOMP_RESULT}" done ```
I don't think we really should care about the compression speed as it's done so rarely the expenses can be neglected.
Ideally we can move buildroot
and images
directories to a separate repository (possibly, even preserving their git history), then set the images up to be built on travis if any of the configs is changed like we do with docker images.
No preference on choice of compression. As long as we don't have to modify defconfigs to decompress, I'm happy. Just pick one; not multiple, if possible.
As a result, I propose that we split the images and buildroot folders off into their own repo
SGTM. Two repos, or one repo? I'm fine with either, but it's unclear to me as stated.
possibly, even preserving their git history
I'm indifferent on this point.
I don't think we really should care about the compression speed as it's done so rarely the expenses can be neglected.
That's a fair point.
Orthogonally, I've been finding it frustrating to:
I think it would be helpful for us to provide:
I don't know what's a good way to organize these, since our CI probably doesn't care about 2-5, but am generally a fan of git submodules (as long as the readme clearly states the correct way to recursively clone, because I always forget to do that). Some can even likely just be fetched lazily (ie. debian images).
In particular, I currently find it difficult to do more userspace testing of clang built kernels, and I think the above would go a long way towards helping. And that doesn't even touch upon packaging kernel modules from a clang built kernel build.
SGTM. Two repos, or one repo? I'm fine with either, but it's unclear to me as stated.
I am planning one repo:
https://github.com/ClangBuiltLinux/boot-utils
https://github.com/nathanchance/boot-utils/commit/1740f54509628e5e2dd5e2c5a01511097b96df73
I will PR when ready for review.
possibly, even preserving their git history
I'm indifferent on this point.
I cannot preserve buildroot
's history with git filter-branch
(link) because of https://github.com/ClangBuiltLinux/continuous-integration/commit/b2848d88fddcaa84e6d0b1473df6f06cf0f6992b and I'd rather not use another tool. We can just link to the history on GitHub like so:
I do not think preserving images
is worthwhile.
I don't think we really should care about the compression speed as it's done so rarely the expenses can be neglected.
Agreed. The balance of decompression speed with regards to ratio is more important.
Orthogonally, I've been finding it frustrating to:
- jump on a new machine without shell history, and try to recreate our QEMU commands from driver.sh. Even sending commands to newbies is a PITA.
- boot more than just our scripts that immediately power off a machine.
Agreed. Maybe we move the QEMU commands into a separate script within the rootfs repo above (and rename it boot-testing
?) so that all we have to do is supply a kernel image and it just boots it for us?
Something like:
./boot-qemu.sh <arch>
and maybe an argument like --shell
for when we want to be dumped into a shell?
EDIT: Renamed the rootfs-images
repo to boot-utils
as this makes life a lot easier.
I think it would be helpful for us to provide:
- userspace images that just powers off the machine
- userspace images that just has a basic shell/toybox
Our images can do both of these. The second one is just done by passing rdinit=/bin/sh
on the command line.
$ qemu-system-x86_64 -append 'console=ttyS0 rdinit=/bin/sh' -cpu host -d unimp,guest_errors -initrd ../../cbl/git/ci-master/images/x86_64/rootfs.cpio -enable-kvm -kernel out/x86_64/arch/x86/boot/bzImage -m 512m -display none -serial mon:stdio
...
/ # mount proc /proc -t proc
/ # cat /proc/version
Linux version 5.6.0-rc7+ (nathan@ubuntu-m2-xlarge-x86) (ClangBuiltLinux clang version 11.0.0 (git://github.com/llvm/llvm-project d264f02c6f502960e2bcdd332f250efc702d09f2)) #1 SMP Thu Mar 26 15:16:14 MST 2020
/ # ls
bin init linuxrc opt run tmp
dev lib media proc sbin usr
etc lib64 mnt root sys var
/ #
- full blown debian image
I do not know that we have the bandwidth for this.
- prebuilt clang built kernel images
- a command line utility that uses some combination of the above
See above.
I don't know what's a good way to organize these, since our CI probably doesn't care about 2-5, but am generally a fan of git submodules (as long as the readme clearly states the correct way to recursively clone, because I always forget to do that). Some can even likely just be fetched lazily (ie. debian images).
Submodules do not play well with Travis unfortunately but cloning the repos and managing them through driver.sh
is not too painful.
In particular, I currently find it difficult to do more userspace testing of clang built kernels, and I think the above would go a long way towards helping. And that doesn't even touch upon packaging kernel modules from a clang built kernel build.
I'll whip something up.
When was zstd decompression support added to the kernel?
It hasn’t been yet: https://lore.kernel.org/lkml/20200325195849.407900-1-nickrterrell@gmail.com/
So I guess lzma
is a better option: faster decompression and support for older kernels.
We can still use zstd
and just decompress the images on the fly before using them. It only takes a maximum of 26ms in my testa above.
That is what I am doing here: https://github.com/nathanchance/boot-utils/blob/updates/boot-qemu.sh#L82-L83
Please review the PR if you have any other concerns, I'll leave it open until Monday evening then merge it: https://github.com/ClangBuiltLinux/boot-utils/pull/1
Images are binary files, and they make up a significant part of the repository size. Each rootfs is 20Mb and each cpio is ~3Mb. Getting more architectures to build means also having more corresponding images. Given that we have 1Gb free LFS storage, maybe we can make use of it.