linuxboot / heads

A minimal Linux that runs as a coreboot or LinuxBoot ROM payload to provide a secure, flexible boot environment for laptops, workstations and servers.
https://osresearch.net/
GNU General Public License v2.0
1.41k stars 186 forks source link

nix buildstack usage not clear enough for end users to use #1736

Closed aluciani closed 1 month ago

aluciani commented 1 month ago

Context of the Build

1. What board are you trying to build? librem_14

2. What repository:branch are you using to build from?

3. What version of coreboot are you trying to build Coreboot-purism which is I think 24.02.01

4. In building the rom where did you get the blobs?

5. If using the automated tools to get the blobs did you run the relevant scripts in the blobs directory

6. What operating system are you using Debian ~12~ 13 (trixie)

Please describe the problem

Describe the bug I'm trying to build the librem_14 board on a debian 12 system. But I get an error in the docker image :

2024-07-27 06:45:16+00:00 MAKE coreboot-purism
tail /home/user/heads/build/x86/log/coreboot-purism.log
-----

Intel ACPI Component Architecture
ASL+ Optimizing Compiler/Disassembler version 20230628
Copyright (c) 2000 - 2023 Intel Corporation

coreboot toolchain v2024-02-18_732134932b
ASL Input:     dsdt.asl -   41461 bytes   1282 keywords      0 source lines
AML Output:    dsdt.aml -   13711 bytes    742 opcodes     540 named objects

Compilation successful. 0 Errors, 0 Warnings, 26 Remarks, 385 Optimizations, 39 Constants Folded
    IASL       3150 warning types were ignored!
    IASL       librem_14/dsdt.aml disassembled correctly.
Makefile:303: *** Missing NSS. Please install libnss3-dev if it is not installed..  Stop.
make[1]: *** [src/security/vboot/Makefile.mk:54: librem_14/external/vboot_reference-ramstage/vboot_fw.a] Error 2
Makefile:303: *** Missing NSS. Please install libnss3-dev if it is not installed..  Stop.
make[1]: *** [src/security/vboot/Makefile.mk:55: librem_14/external/vboot_reference-postcar/vboot_fw.a] Error 2
Makefile:303: *** Missing NSS. Please install libnss3-dev if it is not installed..  Stop.
make[1]: *** [src/security/vboot/Makefile.mk:50: librem_14/external/vboot_reference-bootblock/vboot_fw.a] Error 2
make[1]: Leaving directory '/home/user/heads/build/x86/coreboot-purism'
make: *** [Makefile:563: /home/user/heads/build/x86/coreboot-purism/librem_14/.build] Error 1

To Reproduce Steps to reproduce the behavior:

  1. Install a bare debian ~12~ 13 OS
  2. Follow the docker and nix install on heads README
    $ [ -d /nix ] || sh <(curl -L https://nixos.org/nix/install) --no-daemon
    $ . /home/user/.nix-profile/etc/profile.d/nix.sh
    $ mkdir -p ~/.config/nix
    $ echo 'experimental-features = nix-command flakes' >>~/.config/nix/nix.conf
    $ nix --print-build-logs --verbose build .#dockerImage && docker load < result
  3. run docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=librem_14
  4. See error

Expected behavior Build the librem_14 rom

Additional context I think the board build on CircleCI so I don't know why in local it's not building, maybe libnss3-dev is cached or something ?

srgrint commented 1 month ago

You need to install libnss3-dev . This wasn't needed until the latest coreboot bump yesterday.

aluciani commented 1 month ago

You need to install libnss3-dev . This wasn't needed until the latest coreboot bump yesterday.

yes, but packages installed on debian should not be visible in the docker image, no ?

srgrint commented 1 month ago

You need to install libnss3-dev . This wasn't needed until the latest coreboot bump yesterday.

yes, but packages installed on debian should not be visible in the docker image, no ?

So does it need adding to the flake.nix file?

tlaurion commented 1 month ago

Really confused by this. Seems you are not on master. docker images will create reproducible builds. CircleCI uses versioned docker images (as ov now CircleCI config explicitely tells it uses v0.2.1 where latest points to it)

docker image is build from flake.nix, which contains nss:

user@localhost:~/heads$ cat flake.nix | grep nss
        nss
        openssl # needed for talos-2 kernel build

Or that you have not pulled latest nor v0.2.1 docker image to use docker: user@localhost:~/heads$ time docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=librem_14 [...]

2024-07-27 11:50:31+00:00 INSTALL   build/x86/coreboot-purism/librem_14/coreboot.rom => build/x86/librem_14/heads-librem_14-v0.2.0-2243-g1516320.rom
e8666e27efcabe96e8e7187e62aa227f816eee96299b3d50a72af1ba3efe308f  build/x86/librem_14/heads-librem_14-v0.2.0-2243-g1516320.rom
16777216:build/x86/librem_14/heads-librem_14-v0.2.0-2243-g1516320.rom
rm -rf "/home/user/heads/build/x86/librem_14/update_pkg"
mkdir -p "/home/user/heads/build/x86/librem_14/update_pkg"
cp "/home/user/heads/build/x86/librem_14/heads-librem_14-v0.2.0-2243-g1516320.rom" "/home/user/heads/build/x86/librem_14/update_pkg/"
cd "/home/user/heads/build/x86/librem_14/update_pkg" && sha256sum "heads-librem_14-v0.2.0-2243-g1516320.rom" >sha256sum.txt
cd "/home/user/heads/build/x86/librem_14/update_pkg" && zip -9 "/home/user/heads/build/x86/librem_14/heads-librem_14-v0.2.0-2243-g1516320.zip" "heads-librem_14-v0.2.0-2243-g1516320.rom" sha256sum.txt
  adding: heads-librem_14-v0.2.0-2243-g1516320.rom (deflated 37%)
  adding: sha256sum.txt (deflated 14%)
e8666e27efcabe96e8e7187e62aa227f816eee96299b3d50a72af1ba3efe308f  /home/user/heads/build/x86/librem_14/heads-librem_14-v0.2.0-2243-g1516320.rom
16777216:/home/user/heads/build/x86/librem_14/heads-librem_14-v0.2.0-2243-g1516320.rom

Which latest = v0.2.1 at https://hub.docker.com/r/tlaurion/heads-dev-env/tags: sha256:80c66b6f7a4e69a7c05016fa4756b9755a30146fff8049f3967d4ac27211fa8f

Which both include nss.

What is unclear in README.md?

tlaurion commented 1 month ago

Maybe that's because coreboot-purism per modules/coreboot depends on 24.02.01 buildstack and CircleCI build dependencies builds and cache x230-hotp-maximized which builds 24.02.01 coreboot buildstack first? Repro.

Restarting on clean commit locally.

Using helpers to clean local artifacts:

user@localhost:~/heads$ time docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=librem_14 real.gitclean
----------------------------------------------------------------------
!!!!!! BUILD SYSTEM INFO !!!!!!
System CPUS: 12
System Available Memory: 14843 GB
System Load Average: 0.73
----------------------------------------------------------------------
Used **CPUS**: 12
Used **LOADAVG**: 18
Used **AVAILABLE_MEM_GB**: 14843 GB
----------------------------------------------------------------------
**MAKE_JOBS**: -j12 --load-average=18 

Variables available for override (use 'make VAR_NAME=value'):
**CPUS** (default: number of processors, e.g., 'make CPUS=4')
**LOADAVG** (default: 1.5 times CPUS, e.g., 'make LOADAVG=54')
**AVAILABLE_MEM_GB** (default: memory available on the system in GB, e.g., 'make AVAILABLE_MEM_GB=4')
**MEM_PER_JOB_GB** (default: 1GB per job, e.g., 'make MEM_PER_JOB_GB=2')
----------------------------------------------------------------------
!!!!!! Build starts !!!!!!
Cleaning the repository using Git ignore file as a base...
This will wipe everything not in the Git tree, but keep downloaded coreboot forks (detected as Git repos).
git clean -fxd
Removing blobs/xx30/me.bin
Skipping repository build/ppc64/coreboot-talos_2
Removing build/ppc64/libgcrypt-1.10.1
Removing build/ppc64/util-linux-2.29.2
Removing build/ppc64/slang-2.3.3
Removing build/ppc64/zstd-1.5.5
Removing build/ppc64/npth-1.6
Removing build/ppc64/talos-2
Removing build/ppc64/cryptsetup-2.3.3
Removing build/ppc64/log
Removing build/ppc64/gnupg-2.4.0
Removing build/ppc64/tpmtotp-4d63d21c8b7db2e92ddb393057f168aead147f47
Removing build/ppc64/flashtools-d1e6f12568cb23387144a4b7a6535fe1bc1e79b1
Removing build/ppc64/busybox-1.36.1
Removing build/ppc64/zlib-1.2.11
Removing build/ppc64/kexec-tools-2.0.26
Removing build/ppc64/lvm2.2.02.168
Removing build/ppc64/pinentry-1.1.0
Removing build/ppc64/libgpg-error-1.46
Removing build/ppc64/bash-5.1.16
Removing build/ppc64/linux-5.5
Removing build/ppc64/powerpc-utils-1.3.5
Removing build/ppc64/libusb-1.0.21
Removing build/ppc64/newt-0.52.20
Removing build/ppc64/qrencode-3.4.4
Removing build/ppc64/exfatprogs-1.2.1
Removing build/ppc64/dropbear-2016.74
Removing build/ppc64/libksba-1.6.3
Removing build/ppc64/libassuan-2.5.5
Removing build/ppc64/json-c-0.14
Removing build/ppc64/pciutils-3.5.4
Removing build/ppc64/mbedtls-2.4.2
Removing build/ppc64/popt-1.19
Removing build/ppc64/ncurses-6.5
Removing build/ppc64/musl-cross-38e52db8358c043ae82b346a2e6e66bc86a53bc1
Removing build/ppc64/e2fsprogs-1.47.0
Removing build/ppc64/flashrom-1776bb46ba6ea3d1ab2ec3f0cd88158aabed7400
Skipping repository build/x86/coreboot-purism
Skipping repository build/x86/coreboot-dasharo
Removing build/x86/x220-hotp-maximized
Removing build/x86/libgcrypt-1.10.1
Removing build/x86/libpng-1.6.34
Removing build/x86/x230-maximized
Removing build/x86/w541-hotp-maximized
Removing build/x86/coreboot-4.11
Removing build/x86/UNMAINTAINED_kgpe-d16_server
Removing build/x86/util-linux-2.29.2
Removing build/x86/zstd-1.5.5
Removing build/x86/npth-1.6
Removing build/x86/pixman-0.34.0
Removing build/x86/cairo-1.14.12
Removing build/x86/nitropad-ns50
Removing build/x86/cryptsetup-2.3.3
Removing build/x86/coreboot-24.02.01
Removing build/x86/UNMAINTAINED_kgpe-d16_workstation-usb_keyboard
Removing build/x86/.built
Removing build/x86/librem_11
Removing build/x86/w530-maximized
Removing build/x86/msrtools-572ef8a2b873eda15a322daa48861140a078b92c
Removing build/x86/UNMAINTAINED_kgpe-d16_workstation
Removing build/x86/log
Removing build/x86/t430-hotp-maximized
Removing build/x86/librem_13v4
Removing build/x86/qemu-coreboot-fbwhiptail-tpm2
Removing build/x86/gnupg-2.4.0
Removing build/x86/nitropad-nv41
Removing build/x86/.config
Removing build/x86/w541-maximized
Removing build/x86/tpmtotp-4d63d21c8b7db2e92ddb393057f168aead147f47
Removing build/x86/hotp-verification-e9050e0c914e7a8ffef5d1c82a014e0e2bf79346
Removing build/x86/librem_l1um
Removing build/x86/qemu-coreboot-whiptail-tpm1
Removing build/x86/flashtools-d1e6f12568cb23387144a4b7a6535fe1bc1e79b1
Removing build/x86/busybox-1.36.1
Removing build/x86/librem_15v4
Removing build/x86/qemu-coreboot-fbwhiptail-tpm2-hotp
Removing build/x86/x230-hotp-maximized
Removing build/x86/x230-hotp-legacy
Removing build/x86/zlib-1.2.11
Removing build/x86/t420-hotp-maximized
Removing build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp
Removing build/x86/qemu-coreboot-fbwhiptail-tpm1
Removing build/x86/kexec-tools-2.0.26
Removing build/x86/x230-maximized-fhd_edp
Removing build/x86/x230-hotp-maximized-fhd_edp
Removing build/x86/lvm2.2.02.168
Removing build/x86/pinentry-1.1.0
Removing build/x86/openssl-3.0.8
Removing build/x86/libgpg-error-1.46
Removing build/x86/librem_13v2
Removing build/x86/x230-hotp-maximized_usb-kb
Removing build/x86/UNTESTED_w541-maximized
Removing build/x86/bash-5.1.16
Removing build/x86/x220-maximized
Removing build/x86/qemu-coreboot-whiptail-tpm2-hotp
Removing build/x86/w530-hotp-maximized
Removing build/x86/fbwhiptail-1.3
Removing build/x86/t430-maximized
Removing build/x86/libusb-1.0.21
Removing build/x86/qemu-coreboot-whiptail-tpm2
Removing build/x86/tpm2-tss-3.2.2
Removing build/x86/x230-legacy
Removing build/x86/t530-maximized
Removing build/x86/librem_l1um_v2
Removing build/x86/qrencode-3.4.4
Removing build/x86/t440p-maximized
Removing build/x86/linux-5.10.5
Removing build/x86/librem_15v3
Removing build/x86/UNMAINTAINED_kgpe-d16_server-whiptail
Removing build/x86/exfatprogs-1.2.1
Removing build/x86/dropbear-2016.74
Removing build/x86/t530-hotp-maximized
Removing build/x86/t420-maximized
Removing build/x86/tpm2-tools-5.6
Removing build/x86/z220-cmt-maximized
Removing build/x86/libksba-1.6.3
Removing build/x86/libassuan-2.5.5
Removing build/x86/linux-6.1.8
Removing build/x86/io386-fc73fcf8e51a70638679c3e9b0ada10527f8a7c1
Removing build/x86/json-c-0.14
Removing build/x86/UNTESTED_w541-hotp-maximized
Removing build/x86/librem_mini_v2
Removing build/x86/.build
Removing build/x86/x230-legacy-flash
Removing build/x86/pciutils-3.5.4
Removing build/x86/mbedtls-2.4.2
Removing build/x86/.configured
Removing build/x86/popt-1.19
Removing build/x86/Makefile
Removing build/x86/ncurses-6.5
Removing build/x86/musl-cross-38e52db8358c043ae82b346a2e6e66bc86a53bc1
Removing build/x86/librem_mini
Removing build/x86/e2fsprogs-1.47.0
Removing build/x86/qemu-coreboot-whiptail-tpm1-hotp
Removing build/x86/flashrom-1776bb46ba6ea3d1ab2ec3f0cd88158aabed7400
Removing build/x86/librem_14
Removing config/coreboot-nitropad-ns50.config.old
Removing config/coreboot-nitropad-nv41.config.old
Removing crossgcc/
Removing diff
Removing install/ppc64/
Removing install/x86/
Removing packages/ppc64/
Removing packages/x86/
Removing result

real    0m52.164s
user    0m0.013s
sys 0m0.038s

Note the coreboot clones are not removed, let's remove manually: user@localhost:~/heads$ sudo rm -rf build/x86/coreboot-*

Let's rebuild clean (but with packages/* reused, I save bandwidth): user@localhost:~/heads$ time docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=librem_14

Nope. As expected, clone of coreboot-purism happen, then heads buildstack downloader helper kicks in, enforcing modules/coreboot purism->24.02.01 dependency, which downloads 24.02.01 tarballs and use the release to build coreboot builstack to bootstrap everything first (which requires nss):

user@localhost:~/heads$ time docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=librem_14
----------------------------------------------------------------------
!!!!!! BUILD SYSTEM INFO !!!!!!
System CPUS: 12
System Available Memory: 14867 GB
System Load Average: 0.65
----------------------------------------------------------------------
Used **CPUS**: 12
Used **LOADAVG**: 18
Used **AVAILABLE_MEM_GB**: 14861 GB
----------------------------------------------------------------------
**MAKE_JOBS**: -j12 --load-average=18 

Variables available for override (use 'make VAR_NAME=value'):
**CPUS** (default: number of processors, e.g., 'make CPUS=4')
**LOADAVG** (default: 1.5 times CPUS, e.g., 'make LOADAVG=54')
**AVAILABLE_MEM_GB** (default: memory available on the system in GB, e.g., 'make AVAILABLE_MEM_GB=4')
**MEM_PER_JOB_GB** (default: 1GB per job, e.g., 'make MEM_PER_JOB_GB=2')
----------------------------------------------------------------------
!!!!!! Build starts !!!!!!
if [ ! -e "/home/user/heads/build/x86/coreboot-purism/.canary" ]; then git clone https://source.puri.sm/firmware/coreboot.git "/home/user/heads/build/x86/coreboot-purism"; git -C "/home/user/heads/build/x86/coreboot-purism" reset --hard f4f61b75bcfc50fa3a89f8f438e5312e750f6e3c; echo -n 'https://source.puri.sm/firmware/coreboot.git|f4f61b75bcfc50fa3a89f8f438e5312e750f6e3c' > "/home/user/heads/build/x86/coreboot-purism/.canary"; elif [ "$(cat "/home/user/heads/build/x86/coreboot-purism/.canary")" != 'https://source.puri.sm/firmware/coreboot.git|f4f61b75bcfc50fa3a89f8f438e5312e750f6e3c' ]; then echo "Switching coreboot-purism to https://source.puri.sm/firmware/coreboot.git at f4f61b75bcfc50fa3a89f8f438e5312e750f6e3c" && git -C "/home/user/heads/build/x86/coreboot-purism" reset --hard HEAD^ && echo "git fetch https://source.puri.sm/firmware/coreboot.git f4f61b75bcfc50fa3a89f8f438e5312e750f6e3c --recurse-submodules=no" && git -C "/home/user/heads/build/x86/coreboot-purism" fetch https://source.puri.sm/firmware/coreboot.git f4f61b75bcfc50fa3a89f8f438e5312e750f6e3c --recurse-submodules=no && echo "git reset --hard f4f61b75bcfc50fa3a89f8f438e5312e750f6e3c" && git -C "/home/user/heads/build/x86/coreboot-purism" reset --hard f4f61b75bcfc50fa3a89f8f438e5312e750f6e3c && echo "git clean" && git -C "/home/user/heads/build/x86/coreboot-purism" clean -df && git -C "/home/user/heads/build/x86/coreboot-purism" clean -dffx payloads util/cbmem && echo "git submodule sync" && git -C "/home/user/heads/build/x86/coreboot-purism" submodule sync && echo "git submodule update" && git -C "/home/user/heads/build/x86/coreboot-purism" submodule update --init --checkout && echo -n 'https://source.puri.sm/firmware/coreboot.git|f4f61b75bcfc50fa3a89f8f438e5312e750f6e3c' > "/home/user/heads/build/x86/coreboot-purism/.canary"; fi
Cloning into '/home/user/heads/build/x86/coreboot-purism'...
remote: Enumerating objects: 773214, done.
remote: Counting objects: 100% (623091/623091), done.
remote: Compressing objects: 100% (137287/137287), done.
remote: Total 773214 (delta 562209), reused 507516 (delta 483583), pack-reused 150123 (from 1)
Receiving objects: 100% (773214/773214), 200.45 MiB | 5.51 MiB/s, done.
Resolving deltas: 100% (562227/562227), done.
HEAD is now at f4f61b75bc bootsplash: Increase heap from 1 MB to 4 MB when bootsplash is enabled
if [ ! -e "/home/user/heads/build/x86/coreboot-purism/.patched" ]; then if [ -r patches/coreboot-purism.patch ]; then ( git apply --verbose --reject --binary --directory build/x86/coreboot-purism ) < patches/coreboot-purism.patch || exit 1 ; fi && if [ -d patches/coreboot-purism ] && [ -r patches/coreboot-purism ] ; then for patch in patches/coreboot-purism/*.patch ; do echo "Applying patch file : $patch " ; ( git apply --verbose --reject --binary --directory build/x86/coreboot-purism ) < $patch || exit 1 ; done ; fi && touch "/home/user/heads/build/x86/coreboot-purism/.patched"; fi
2024-07-27 12:08:54+00:00 WGET https://www.coreboot.org/releases/coreboot-24.02.01.tar.xz
--2024-07-27 12:08:54--  https://www.coreboot.org/releases/coreboot-24.02.01.tar.xz
Resolving www.coreboot.org (www.coreboot.org)... 78.46.105.101, 2a01:4f8:121:1254::2
Connecting to www.coreboot.org (www.coreboot.org)|78.46.105.101|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 68456284 (65M) [application/octet-stream]
Saving to: '/home/user/heads/packages/x86/coreboot-24.02.01.tar.xz.tmp'

/home/user/heads/packages/x86/coreboot-24.02.01.tar. 100%[=====================================================================================================================>]  65.28M  4.62MB/s    in 28s     

2024-07-27 12:09:23 (2.34 MB/s) - '/home/user/heads/packages/x86/coreboot-24.02.01.tar.xz.tmp' saved [68456284/68456284]

/home/user/heads/packages/x86/coreboot-24.02.01.tar.xz.tmp: OK
mkdir -p "/home/user/heads/build/x86/coreboot-24.02.01/"
tar -xf "/home/user/heads/packages/x86/coreboot-24.02.01.tar.xz" --strip 1 -C "/home/user/heads/build/x86/coreboot-24.02.01/"
if [ -r patches/coreboot-24.02.01.patch ]; then ( git apply --verbose --reject --binary --directory build/x86/coreboot-24.02.01 ) < patches/coreboot-24.02.01.patch || exit 1 ; fi
if [ -d patches/coreboot-24.02.01 ] && [ -r patches/coreboot-24.02.01 ] ; then for patch in patches/coreboot-24.02.01/*.patch ; do echo "Applying patch file : $patch " ; ( git apply --verbose --reject --binary --directory build/x86/coreboot-24.02.01 ) < $patch || exit 1 ; done ; fi
Applying patch file : patches/coreboot-24.02.01/0001-prevent_bootsplash_heap_allocation_failure_to_booting.patch 
Checking patch build/x86/coreboot-24.02.01/src/lib/malloc.c...
Applied patch build/x86/coreboot-24.02.01/src/lib/malloc.c cleanly.
Applying patch file : patches/coreboot-24.02.01/0002-increase_heap_from1mb_to4mb_when_bootpslash_enabled.patch 
Checking patch build/x86/coreboot-24.02.01/src/device/Kconfig...
Applied patch build/x86/coreboot-24.02.01/src/device/Kconfig cleanly.
WGET="" bin/fetch_coreboot_crossgcc_archive.sh "/home/user/heads/build/x86/coreboot-24.02.01" "gmp" "/home/user/heads/packages/x86"
--2024-07-27 12:09:27--  https://ftpmirror.gnu.org/gmp/gmp-6.3.0.tar.xz
Resolving ftpmirror.gnu.org (ftpmirror.gnu.org)... 209.51.188.200, 2001:470:142:5::200
Connecting to ftpmirror.gnu.org (ftpmirror.gnu.org)|209.51.188.200|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://mirror.csclub.uwaterloo.ca/gnu/gmp/gmp-6.3.0.tar.xz [following]
--2024-07-27 12:09:29--  https://mirror.csclub.uwaterloo.ca/gnu/gmp/gmp-6.3.0.tar.xz
Resolving mirror.csclub.uwaterloo.ca (mirror.csclub.uwaterloo.ca)... 129.97.134.71, 2620:101:f000:4901:c5c:0:f:1055
Connecting to mirror.csclub.uwaterloo.ca (mirror.csclub.uwaterloo.ca)|129.97.134.71|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2094196 (2.0M) [text/plain]
Saving to: '/home/user/heads/packages/x86/coreboot-crossgcc-gmp-6.3.0.tar.xz.tmp'

/home/user/heads/packages/x86/coreboot-crossgcc-gmp- 100%[=====================================================================================================================>]   2.00M  4.11MB/s    in 0.5s    

2024-07-27 12:09:30 (4.11 MB/s) - '/home/user/heads/packages/x86/coreboot-crossgcc-gmp-6.3.0.tar.xz.tmp' saved [2094196/2094196]

/home/user/heads/packages/x86/coreboot-crossgcc-gmp-6.3.0.tar.xz.tmp: OK
touch "/home/user/heads/build/x86/coreboot-24.02.01/.heads-crossgcc-pkg-gmp"
WGET="" bin/fetch_coreboot_crossgcc_archive.sh "/home/user/heads/build/x86/coreboot-24.02.01" "mpfr" "/home/user/heads/packages/x86"
--2024-07-27 12:09:30--  https://ftpmirror.gnu.org/mpfr/mpfr-4.2.1.tar.xz
Resolving ftpmirror.gnu.org (ftpmirror.gnu.org)... 209.51.188.200, 2001:470:142:5::200
Connecting to ftpmirror.gnu.org (ftpmirror.gnu.org)|209.51.188.200|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://mirror2.evolution-host.com/gnu/mpfr/mpfr-4.2.1.tar.xz [following]
--2024-07-27 12:09:30--  https://mirror2.evolution-host.com/gnu/mpfr/mpfr-4.2.1.tar.xz
Resolving mirror2.evolution-host.com (mirror2.evolution-host.com)... 167.114.8.249, 2607:5300:60:450d:c259:13f4:6df0:1
Connecting to mirror2.evolution-host.com (mirror2.evolution-host.com)|167.114.8.249|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1493608 (1.4M) [application/x-xz]
Saving to: '/home/user/heads/packages/x86/coreboot-crossgcc-mpfr-4.2.1.tar.xz.tmp'

/home/user/heads/packages/x86/coreboot-crossgcc-mpfr 100%[=====================================================================================================================>]   1.42M  4.83MB/s    in 0.3s    

2024-07-27 12:09:31 (4.83 MB/s) - '/home/user/heads/packages/x86/coreboot-crossgcc-mpfr-4.2.1.tar.xz.tmp' saved [1493608/1493608]

/home/user/heads/packages/x86/coreboot-crossgcc-mpfr-4.2.1.tar.xz.tmp: OK
touch "/home/user/heads/build/x86/coreboot-24.02.01/.heads-crossgcc-pkg-mpfr"
WGET="" bin/fetch_coreboot_crossgcc_archive.sh "/home/user/heads/build/x86/coreboot-24.02.01" "mpc" "/home/user/heads/packages/x86"
--2024-07-27 12:09:31--  https://ftpmirror.gnu.org/mpc/mpc-1.3.1.tar.gz
Resolving ftpmirror.gnu.org (ftpmirror.gnu.org)... 209.51.188.200, 2001:470:142:5::200
Connecting to ftpmirror.gnu.org (ftpmirror.gnu.org)|209.51.188.200|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://mirror.its.dal.ca/gnu/mpc/mpc-1.3.1.tar.gz [following]
--2024-07-27 12:09:31--  https://mirror.its.dal.ca/gnu/mpc/mpc-1.3.1.tar.gz
Resolving mirror.its.dal.ca (mirror.its.dal.ca)... 192.75.96.254
Connecting to mirror.its.dal.ca (mirror.its.dal.ca)|192.75.96.254|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 773573 (755K) [application/octet-stream]
Saving to: '/home/user/heads/packages/x86/coreboot-crossgcc-mpc-1.3.1.tar.gz.tmp'

/home/user/heads/packages/x86/coreboot-crossgcc-mpc- 100%[=====================================================================================================================>] 755.44K  2.89MB/s    in 0.3s    

2024-07-27 12:09:32 (2.89 MB/s) - '/home/user/heads/packages/x86/coreboot-crossgcc-mpc-1.3.1.tar.gz.tmp' saved [773573/773573]

/home/user/heads/packages/x86/coreboot-crossgcc-mpc-1.3.1.tar.gz.tmp: OK
touch "/home/user/heads/build/x86/coreboot-24.02.01/.heads-crossgcc-pkg-mpc"
WGET="" bin/fetch_coreboot_crossgcc_archive.sh "/home/user/heads/build/x86/coreboot-24.02.01" "binutils" "/home/user/heads/packages/x86"
--2024-07-27 12:09:32--  https://ftpmirror.gnu.org/binutils/binutils-2.41.tar.xz
Resolving ftpmirror.gnu.org (ftpmirror.gnu.org)... 209.51.188.200, 2001:470:142:5::200
Connecting to ftpmirror.gnu.org (ftpmirror.gnu.org)|209.51.188.200|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://mirror.its.dal.ca/gnu/binutils/binutils-2.41.tar.xz [following]
--2024-07-27 12:09:32--  https://mirror.its.dal.ca/gnu/binutils/binutils-2.41.tar.xz
Resolving mirror.its.dal.ca (mirror.its.dal.ca)... 192.75.96.254
Connecting to mirror.its.dal.ca (mirror.its.dal.ca)|192.75.96.254|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26765692 (26M) [application/octet-stream]
Saving to: '/home/user/heads/packages/x86/coreboot-crossgcc-binutils-2.41.tar.xz.tmp'

/home/user/heads/packages/x86/coreboot-crossgcc-binu 100%[=====================================================================================================================>]  25.53M  5.54MB/s    in 5.0s    

2024-07-27 12:09:37 (5.15 MB/s) - '/home/user/heads/packages/x86/coreboot-crossgcc-binutils-2.41.tar.xz.tmp' saved [26765692/26765692]

/home/user/heads/packages/x86/coreboot-crossgcc-binutils-2.41.tar.xz.tmp: OK
touch "/home/user/heads/build/x86/coreboot-24.02.01/.heads-crossgcc-pkg-binutils"
WGET="" bin/fetch_coreboot_crossgcc_archive.sh "/home/user/heads/build/x86/coreboot-24.02.01" "gcc" "/home/user/heads/packages/x86"
--2024-07-27 12:09:38--  https://ftpmirror.gnu.org/gcc/gcc-13.2.0/gcc-13.2.0.tar.xz
Resolving ftpmirror.gnu.org (ftpmirror.gnu.org)... 209.51.188.200, 2001:470:142:5::200
Connecting to ftpmirror.gnu.org (ftpmirror.gnu.org)|209.51.188.200|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://mirror.csclub.uwaterloo.ca/gnu/gcc/gcc-13.2.0/gcc-13.2.0.tar.xz [following]
--2024-07-27 12:09:38--  https://mirror.csclub.uwaterloo.ca/gnu/gcc/gcc-13.2.0/gcc-13.2.0.tar.xz
Resolving mirror.csclub.uwaterloo.ca (mirror.csclub.uwaterloo.ca)... 129.97.134.71, 2620:101:f000:4901:c5c:0:f:1055
Connecting to mirror.csclub.uwaterloo.ca (mirror.csclub.uwaterloo.ca)|129.97.134.71|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 87858592 (84M) [text/plain]
Saving to: '/home/user/heads/packages/x86/coreboot-crossgcc-gcc-13.2.0.tar.xz.tmp'

/home/user/heads/packages/x86/coreboot-crossgcc-gcc- 100%[=====================================================================================================================>]  83.79M  4.28MB/s    in 24s     

2024-07-27 12:10:04 (3.45 MB/s) - '/home/user/heads/packages/x86/coreboot-crossgcc-gcc-13.2.0.tar.xz.tmp' saved [87858592/87858592]

/home/user/heads/packages/x86/coreboot-crossgcc-gcc-13.2.0.tar.xz.tmp: OK
touch "/home/user/heads/build/x86/coreboot-24.02.01/.heads-crossgcc-pkg-gcc"
WGET="" bin/fetch_coreboot_crossgcc_archive.sh "/home/user/heads/build/x86/coreboot-24.02.01" "nasm" "/home/user/heads/packages/x86"
--2024-07-27 12:10:04--  https://www.nasm.us/pub/nasm/releasebuilds/2.16.01/nasm-2.16.01.tar.bz2
Resolving www.nasm.us (www.nasm.us)... 198.137.202.136, 2607:7c80:54:3::136
Connecting to www.nasm.us (www.nasm.us)|198.137.202.136|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1351571 (1.3M) [application/x-bzip2]
Saving to: '/home/user/heads/packages/x86/coreboot-crossgcc-nasm-2.16.01.tar.bz2.tmp'

/home/user/heads/packages/x86/coreboot-crossgcc-nasm 100%[=====================================================================================================================>]   1.29M  2.37MB/s    in 0.5s    

2024-07-27 12:10:05 (2.37 MB/s) - '/home/user/heads/packages/x86/coreboot-crossgcc-nasm-2.16.01.tar.bz2.tmp' saved [1351571/1351571]

/home/user/heads/packages/x86/coreboot-crossgcc-nasm-2.16.01.tar.bz2.tmp: OK
touch "/home/user/heads/build/x86/coreboot-24.02.01/.heads-crossgcc-pkg-nasm"
WGET="" bin/fetch_coreboot_crossgcc_archive.sh "/home/user/heads/build/x86/coreboot-24.02.01" "iasl" "/home/user/heads/packages/x86"
--2024-07-27 12:10:05--  https://github.com/acpica/acpica/archive/refs/tags/R06_28_23.tar.gz
Resolving github.com (github.com)... 140.82.114.4
Connecting to github.com (github.com)|140.82.114.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/acpica/acpica/tar.gz/refs/tags/R06_28_23 [following]
--2024-07-27 12:10:06--  https://codeload.github.com/acpica/acpica/tar.gz/refs/tags/R06_28_23
Resolving codeload.github.com (codeload.github.com)... 140.82.113.10
Connecting to codeload.github.com (codeload.github.com)|140.82.113.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: '/home/user/heads/packages/x86/coreboot-crossgcc-R06_28_23.tar.gz.tmp'

/home/user/heads/packages/x86/coreboot-crossgcc-R06_     [        <=>                                                                                                           ]   7.16M  4.81MB/s    in 1.5s    

2024-07-27 12:10:07 (4.81 MB/s) - '/home/user/heads/packages/x86/coreboot-crossgcc-R06_28_23.tar.gz.tmp' saved [7505765]

/home/user/heads/packages/x86/coreboot-crossgcc-R06_28_23.tar.gz.tmp: OK
touch "/home/user/heads/build/x86/coreboot-24.02.01/.heads-crossgcc-pkg-iasl"
make -C "/home/user/heads/build/x86/coreboot-24.02.01" CPUS=12 "crossgcc-i386" \

make[1]: Entering directory '/home/user/heads/build/x86/coreboot-24.02.01'
Welcome to the coreboot cross toolchain builder v_ 

Building toolchain using 12 thread(s).

Target architecture is i386-elf

Found compatible Ada compiler, enabling Ada support by default.

Downloading and verifying tarballs ...
 * gmp-6.3.0.tar.xz (cached)... hash verified (b4043dd2964ab1a858109da85c44de224384f352)
 * mpfr-4.2.1.tar.xz (cached)... hash verified (31ffb4244cb469e2b4937cce1f50150300971dfb)
 * mpc-1.3.1.tar.gz (cached)... hash verified (bac1c1fa79f5602df1e29e4684e103ad55714e02)
 * binutils-2.41.tar.xz (cached)... hash verified (0e008260a958bbd10182ee3384672ae0a310eece)
 * gcc-13.2.0.tar.xz (cached)... hash verified (5f95b6d042fb37d45c6cbebfc91decfbc4fb493c)
Downloaded tarballs ... ok
Unpacking and patching ...
 * gmp-6.3.0.tar.xz
   o gmp-6.3.0_generic-build.patch
 * mpfr-4.2.1.tar.xz
 * mpc-1.3.1.tar.gz
 * binutils-2.41.tar.xz
   o binutils-2.41_as-ipxe.patch
   o binutils-2.41_no-makeinfo.patch
 * gcc-13.2.0.tar.xz
   o gcc-13.2.0_asan_shadow_offset_callback.patch
   o gcc-13.2.0_gnat.patch
   o gcc-13.2.0_libcpp.patch
   o gcc-13.2.0_libgcc.patch
   o gcc-13.2.0_musl_poisoned_calloc.patch
   o gcc-13.2.0_rv32iafc.patch
Unpacked and patched ... ok
Building packages ...
Building GMP v6.3.0 for host ... 

For those interested, those helpers are under bin/fetch_source_archive.sh and bin/fetch_coreboot_crossgcc_archive.sh which global Makefile enforces usage so that tarballs can be downloaded from mirrors since they have a tendency to impact Heads users, thanks to Purism for caching them on their mirror, which are validated per original logic verifying hashes before extraction both under Heads modules/* and coreboot buildstack...

user@localhost:~/heads$ grep -R archive.sh Makefile modules/ 
Makefile:       WGET="$(WGET)" bin/fetch_source_archive.sh "$($1_url)" "$$@" "$($1_hash)"
modules/coreboot:# fetch_source_archive.sh to leverage our mirrors.
modules/coreboot:   WGET="$(WGET)" bin/fetch_coreboot_crossgcc_archive.sh \

Cannot replicate: user error and or documentation unclear, please do a PR once successful reproducing @aluciani ! @srgrint seems like you haven't followed nix buildstack changes under Heads, please read and review README.md and do PR, will review!

tlaurion commented 1 month ago

Renamed issue since this is README.md nix docker image replication/nix based docker image replication that is unclear, and not a bug with the buildsystem itself to not let others in the belief that things are broken outside of replicability of README.md instructions to use nix and flakes.nix, or docker usage and git cloning, or using git pointing at master, not sure.

Going AFK, beautiful outside, crisis averted.


After all, if CircleCI can build, end users should too. If not, PR against doc welcome peaople: Documentation is community effort, thank you.

tlaurion commented 1 month ago

https://osresearch.net/general-building/ -> https://github.com/linuxboot/heads-wiki/edit/master/Installing-and-Configuring/Building-Heads/general.md

And

https://github.com/linuxboot/heads/blob/master/README.md#pull-docker-hub-image-to-prepare-reproducible-roms-as-circleci-in-one-call

Things clear to my eyes, but as usual, this is not a guarantee its clear to yours so please modify so that clear to yours @aluciani @srgrint :)

aluciani commented 1 month ago
$ [ -d /nix ] || sh <(curl -L https://nixos.org/nix/install) --no-daemon
$ . /home/user/.nix-profile/etc/profile.d/nix.sh
$ mkdir -p ~/.config/nix
$ echo 'experimental-features = nix-command flakes' >>~/.config/nix/nix.conf
$ sudo rm -rf ./heads
$ git clone https://github.com/linuxboot/heads
$ docker -v
Docker version 27.1.1, build 6312585
$ nix --version
nix (Nix) 2.23.3

$ docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=librem_14

....

tail /home/user/heads/build/x86/log/coreboot-purism.log
-----
    CP         ramstage/lib/program.ld
    GCC        ramstage/libgfxinit/common/dyncpu/hw-gfx-gma-config.o
    GCC        ramstage/libgfxinit/common/haswell_shared/hw-gfx-gma-connectors-ddi.o
    GCC        ramstage/libgfxinit/common/haswell_shared/hw-gfx-gma-connectors.o
    GCC        ramstage/libgfxinit/common/haswell_shared/hw-gfx-gma-ddi_phy_stub.o
    GCC        ramstage/libgfxinit/common/haswell_shared/hw-gfx-gma-port_detect.o
    GCC        ramstage/libgfxinit/common/hw-gfx-dp_aux_ch.o
    GCC        ramstage/libgfxinit/common/hw-gfx-dp_defs.o
    GCC        ramstage/libgfxinit/common/hw-gfx-dp_dual_mode.o
    GCC        ramstage/libgfxinit/common/hw-gfx-dp_info.o
Makefile:303: *** Missing NSS. Please install libnss3-dev if it is not installed..  Stop.
make[1]: *** [src/security/vboot/Makefile.mk:52: librem_14/external/vboot_reference-romstage/vboot_fw.a] Error 2
make[1]: *** Waiting for unfinished jobs....
Makefile:303: *** Missing NSS. Please install libnss3-dev if it is not installed..  Stop.
make[1]: *** [src/security/vboot/Makefile.mk:54: librem_14/external/vboot_reference-ramstage/vboot_fw.a] Error 2
Makefile:303: *** Missing NSS. Please install libnss3-dev if it is not installed..  Stop.
make[1]: *** [src/security/vboot/Makefile.mk:55: librem_14/external/vboot_reference-postcar/vboot_fw.a] Error 2
Makefile:303: *** Missing NSS. Please install libnss3-dev if it is not installed..  Stop.
make[1]: *** [src/security/vboot/Makefile.mk:50: librem_14/external/vboot_reference-bootblock/vboot_fw.a] Error 2
make[1]: Leaving directory '/home/user/heads/build/x86/coreboot-purism'
make: *** [Makefile:563: /home/user/heads/build/x86/coreboot-purism/librem_14/.build] Error 1

I MADE A MISTAKE ON THE OS, I M USING DEBIAN TRIXIE (not that it should change something on the build......)

$ uname -a
Linux workstation 6.9.10-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.9.10-1 (2024-07-19) x86_64 GNU/Linux
tlaurion commented 1 month ago
$ [ -d /nix ] || sh <(curl -L https://nixos.org/nix/install) --no-daemon
$ . /home/user/.nix-profile/etc/profile.d/nix.sh
$ mkdir -p ~/.config/nix
$ echo 'experimental-features = nix-command flakes' >>~/.config/nix/nix.conf
$ sudo rm -rf ./heads
$ git clone https://github.com/linuxboot/heads
$ docker -v
Docker version 27.1.1, build 6312585
$ nix --version
nix (Nix) 2.23.3

$ docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=librem_14

....

tail /home/user/heads/build/x86/log/coreboot-purism.log
-----
    CP         ramstage/lib/program.ld
    GCC        ramstage/libgfxinit/common/dyncpu/hw-gfx-gma-config.o
    GCC        ramstage/libgfxinit/common/haswell_shared/hw-gfx-gma-connectors-ddi.o
    GCC        ramstage/libgfxinit/common/haswell_shared/hw-gfx-gma-connectors.o
    GCC        ramstage/libgfxinit/common/haswell_shared/hw-gfx-gma-ddi_phy_stub.o
    GCC        ramstage/libgfxinit/common/haswell_shared/hw-gfx-gma-port_detect.o
    GCC        ramstage/libgfxinit/common/hw-gfx-dp_aux_ch.o
    GCC        ramstage/libgfxinit/common/hw-gfx-dp_defs.o
    GCC        ramstage/libgfxinit/common/hw-gfx-dp_dual_mode.o
    GCC        ramstage/libgfxinit/common/hw-gfx-dp_info.o
Makefile:303: *** Missing NSS. Please install libnss3-dev if it is not installed..  Stop.
make[1]: *** [src/security/vboot/Makefile.mk:52: librem_14/external/vboot_reference-romstage/vboot_fw.a] Error 2
make[1]: *** Waiting for unfinished jobs....
Makefile:303: *** Missing NSS. Please install libnss3-dev if it is not installed..  Stop.
make[1]: *** [src/security/vboot/Makefile.mk:54: librem_14/external/vboot_reference-ramstage/vboot_fw.a] Error 2
Makefile:303: *** Missing NSS. Please install libnss3-dev if it is not installed..  Stop.
make[1]: *** [src/security/vboot/Makefile.mk:55: librem_14/external/vboot_reference-postcar/vboot_fw.a] Error 2
Makefile:303: *** Missing NSS. Please install libnss3-dev if it is not installed..  Stop.
make[1]: *** [src/security/vboot/Makefile.mk:50: librem_14/external/vboot_reference-bootblock/vboot_fw.a] Error 2
make[1]: Leaving directory '/home/user/heads/build/x86/coreboot-purism'
make: *** [Makefile:563: /home/user/heads/build/x86/coreboot-purism/librem_14/.build] Error 1

I MADE A MISTAKE ON THE OS, I M USING DEBIAN TRIXIE (not that it should change something on the build......)

$ uname -a
Linux workstation 6.9.10-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.9.10-1 (2024-07-19) x86_64 GNU/Linux

Exactly. It shouldn't matter if you actually constructed the docker image (which your repro doesn't do above) by continuing following steps from master commit see my repo logs, you miss doing

https://github.com/linuxboot/heads/tree/1519bd74fe5aa16bb930165a8649cdfe45b8f6dc?tab=readme-ov-file

I see no

nix --print-build-logs --verbose develop --ignore-environment --command true

nix --print-build-logs --verbose build .#dockerImage && docker load < result

Which outputs nix at result and then construct the docker image. You are basically reusing your old self created docker image from an old nix flake toy created in the past

Then https://github.com/linuxboot/heads/tree/1519bd74fe5aa16bb930165a8649cdfe45b8f6dc?tab=readme-ov-file#jump-into-nix-develop-created-docker-image-for-interactive-workflow

docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) linuxboot/heads:dev-env -- make BOARD=nitropad-nv41

Which are all steps to create reproducible docker image from nix flakes and then use it to build a board roms which is developer oriented which you should not have to do.


The only thing you need to do is at https://github.com/linuxboot/heads/tree/1519bd74fe5aa16bb930165a8649cdfe45b8f6dc?tab=readme-ov-file#pull-docker-hub-image-to-prepare-reproducible-roms-as-circleci-in-one-call

Which uses recreated docker image for end users (above prior steps are for developers not end users)

END USER SHOULD ONLY DO : docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=x230-hotp-maximized

Replacing xyz with proper board at BOARD=xyz docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=nitropad-nv41


@aluciani at this point, I expect a PR because clearly the documebtariin was not clear enough to you, the end user, not the developer.

Thanks in advance for creating such PR.

tlaurion commented 1 month ago

@aluciani : Tldr you are reusing past self-created docker image from old flake.nix that was not containing nss.

Read prior edited comment. If going that path, you cannot skip steps.

tlaurion commented 1 month ago

@aluciani do

docker image list docker image rm IMAGE_ID_FROM_PREVIOUS_COMMAND

This weird situation happens if "latest" matches locally built IMAGE_ID from reproducible builds. For users only using latest, this is not an issue. For people recreating docker image from "result" produce nix image, this is not an issue either.

I just ran in the same issue on an old dev qube that was using docker image v0.1.9 on which I was trying to make qemu target and failed with similar results because nss was not in docker image v0.1.9.

calling docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=qemu-coreboot-whiptail-tpm1

built as expected.


Now, how should this be documented properly?

aluciani commented 1 month ago

First of all I apologize, I'm a stupid sandwich, I hadn't read the documentation properly. It's clear and I don't think it needs updating on my part (at least for now). I'm sorry for the delay in replying, I'm not very available at the moment, and the build is taking a long time. Anyway, I've just launched it with all the advice (delete the docker image etc). Do you know if, when I run the command :

docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=whatever

if the docker image is not up to date, is it updated automatically? Anyway, after deleting the docker image, re pull, I get this :

$ docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=librem_14

...

2024-07-29 14:02:47+00:00 MAKE coreboot-purism
tail /home/user/heads/build/x86/log/coreboot-purism.log
-----
    CC         postcar/soc/intel/common/block/cpu/car/exit_car.o
    CC+STRIP   src/cpu/intel/fit/fit_table.c
    HOSTCC     cbfstool/cbfstool.o
    HOSTCC     cbfstool/cbfs_image.o
    HOSTCC     cbfstool/cbfs-mkstage.o
    HOSTCC     cbfstool/cbfs-mkpayload.o
    HOSTCC     cbfstool/partitioned_file.o
    HOSTCC     cbfstool/platform_fixups.o
convert: insufficient image data in file `../../../branding/Heads/bootsplash.jpg' @ error/jpeg.c/ReadOneJPEGImage/1136.
convert: no images defined `bmp:librem_14/mainboard/purism/librem_cnl/cbfs-file.2pZPsr.out' @ error/convert.c/ConvertImageCommand/3362.
convert: unable to open image 'librem_14/mainboard/purism/librem_cnl/cbfs-file.2pZPsr.out': No such file or directory @ error/blob.c/OpenBlob/3596.
convert: no decode delegate for this image format `OUT' @ error/constitute.c/ReadImage/746.
convert: no images defined `jpg:librem_14/mainboard/purism/librem_cnl/cbfs-file.2pZPsr.out' @ error/convert.c/ConvertImageCommand/3362.
make[1]: *** [Makefile:315: librem_14/mainboard/purism/librem_cnl/cbfs-file.2pZPsr.out] Error 1
make[1]: *** Waiting for unfinished jobs....
vboot SHA256 built with tight loops (slower, smaller code size)
vboot SHA256 built with tight loops (slower, smaller code size)
vboot SHA256 built with tight loops (slower, smaller code size)
vboot SHA256 built with tight loops (slower, smaller code size)
make[1]: Leaving directory '/home/user/heads/build/x86/coreboot-purism'
make: *** [Makefile:563: /home/user/heads/build/x86/coreboot-purism/librem_14/.build] Error 1

I think this is a symlink issue, no ?

tlaurion commented 1 month ago

First of all I apologize, I'm a stupid sandwich, I hadn't read the documentation properly. It's clear and I don't think it needs updating on my part (at least for now). I'm sorry for the delay in replying, I'm not very available at the moment, and the build is taking a long time. Anyway, I've just launched it with all the advice (delete the docker image etc). Do you know if, when I run the command :

docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=whatever

if the docker image is not up to date, is it updated automatically? Anyway, after deleting the docker image, re pull, I get this :

It is updated automatically if you haven't created and tagged your own locally created docker image out of nix output being "result". I'm sorry I should have outputted my docker images list output so that you can visually understand what i'm trying to express in words, but on my side, I had 3 docker images having the same hashes, with me having created tagged that linked them all together.

TLDR: if you pulled latest docker image from dockerhub, on next call with latest, docker will check if its computed checksum is still the same and if not will pull latest layers changes. But if you created a latest image yourself, it won't update, which is what happened to you.

Therefore, unless you have a reproducibility issue or attempting to address https://github.com/linuxboot/heads/discussions/1690 discussion, you should not have to build the docker image yourself unless your goal is to add more stuff into flake.nix; you should either use same docker image version specified under CircleCI config (v0.2.1 as of today under master https://github.com/linuxboot/heads/blob/2ea14bcbeb6ed6d5ddba7429a2260f8c1417c243/.circleci/config.yml#L48 or latest, which I should always sync after a PR adding stuff inside of the docker image just like history of flake.nix should inform you of which PR changed it, see https://github.com/linuxboot/heads/commits/master/flake.nix. Building docker image is a developer related step, not end user. End users should use prebuilt docker images, just like CI and developers should replicate the docker image and report of reproducibility effort. It needs a bit of understanding of what happens in the background to be able to do integrity/authenticity validation; I think this issue had enough e-ink spread and should focus on improving the doc if needed, I would love to not answer these kind of questions by other users.

Therefore I ask you to suggest improvements to docs to keep my sanity, please.

$ docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=librem_14

...

2024-07-29 14:02:47+00:00 MAKE coreboot-purism
tail /home/user/heads/build/x86/log/coreboot-purism.log
-----
    CC         postcar/soc/intel/common/block/cpu/car/exit_car.o
    CC+STRIP   src/cpu/intel/fit/fit_table.c
    HOSTCC     cbfstool/cbfstool.o
    HOSTCC     cbfstool/cbfs_image.o
    HOSTCC     cbfstool/cbfs-mkstage.o
    HOSTCC     cbfstool/cbfs-mkpayload.o
    HOSTCC     cbfstool/partitioned_file.o
    HOSTCC     cbfstool/platform_fixups.o
convert: insufficient image data in file `../../../branding/Heads/bootsplash.jpg' @ error/jpeg.c/ReadOneJPEGImage/1136.
convert: no images defined `bmp:librem_14/mainboard/purism/librem_cnl/cbfs-file.2pZPsr.out' @ error/convert.c/ConvertImageCommand/3362.
convert: unable to open image 'librem_14/mainboard/purism/librem_cnl/cbfs-file.2pZPsr.out': No such file or directory @ error/blob.c/OpenBlob/3596.
convert: no decode delegate for this image format `OUT' @ error/constitute.c/ReadImage/746.
convert: no images defined `jpg:librem_14/mainboard/purism/librem_cnl/cbfs-file.2pZPsr.out' @ error/convert.c/ConvertImageCommand/3362.
make[1]: *** [Makefile:315: librem_14/mainboard/purism/librem_cnl/cbfs-file.2pZPsr.out] Error 1
make[1]: *** Waiting for unfinished jobs....
vboot SHA256 built with tight loops (slower, smaller code size)
vboot SHA256 built with tight loops (slower, smaller code size)
vboot SHA256 built with tight loops (slower, smaller code size)
vboot SHA256 built with tight loops (slower, smaller code size)
make[1]: Leaving directory '/home/user/heads/build/x86/coreboot-purism'
make: *** [Makefile:563: /home/user/heads/build/x86/coreboot-purism/librem_14/.build] Error 1

I think this is a symlink issue, no ?

Can you please just start clean, replicate all steps from the documentation? IF CircleCI is able to build roms, you should too. I cannot just play mind games to imagine which commands you missed trying to replicate things that you can't on a system that is made to be reproducible.

Start clean, wipe heads, clone it again, follow README.md instructions steps, each of them, and then have my brain time be used efficiently please. This is not happening at https://github.com/linuxboot/heads/commit/2ea14bcbeb6ed6d5ddba7429a2260f8c1417c243 checkmark building librem_14 with docker image v0.2.1 : https://app.circleci.com/pipelines/github/linuxboot/heads/823/workflows/750726e8-5afe-4fff-9160-1d747dc9b36f/jobs/18306

And me having rebuilt locally with latest docker image previously already showed steps to restart building clean on same repo and where to look at to understand helpers and build systems; there is nothing much more I can do here to help but state maybe you are not understanding enough how things work, unfortunately, which would need to have documentation improved from developers to end users, and It seems i'm not able to brain dump and eli5 to end users enough how to start clean outside of starting clean and obtain the same results.

Your error in your last reply shows you have previous builds artifacts in your way, and since I cannot reproduce, I cannot invest too much time in this and can only recommend starting clean from a new git clone from official docs and then reproduce from README.md steps and close this issue.

If your builds take too long, you can look for helpers that were created and documented in code under global Makefile, and search PR documentation under github: I already put a lot of time and effort there documenting as code (for developers) and git blame should help you find PR and discussions and need that lead to create such helpers, eg: https://github.com/linuxboot/heads/blame/2ea14bcbeb6ed6d5ddba7429a2260f8c1417c243/Makefile#L874

tlaurion commented 1 month ago

As for your symlink issue, yes branding/Heads/bootsplash.jpg is a symlink that you might have broken, but if so, another user issue which cannot be reproduced and would deserve redoing symlink, and unrelated to the original issue opened here. A clean git clone would also show you how things are supposed to work. coreboot config embeds a path to the directory of bootsplash.jpg with a static name, and bootsplash.jpg is a symlink to a bootsplash. Its easier to change a symlink then a config file; therefore the path I chose to implement, where forks can decide for their own what is best for them and maintain the way then intend to do. IF this is an issue, open another one and work with me keeping things seperated.

If this issue is fixed, please close it @aluciani