Open osresearch opened 3 years ago
Excluding the musl-cross
build, a normal build executes these programs this many times:
19091 /usr/bin/sed
13614 /usr/bin/rm
12855 /root/heads/crossgcc/bin/x86_64-linux-musl-gcc
12300 /root/heads/crossgcc/bin/../libexec/gcc/x86_64-linux-musl/8.3.0/cc1
11125 /bin/sh
9889 /usr/bin/cat
7518 /bin/bash
7093 /root/heads/crossgcc/bin/../lib/gcc/x86_64-linux-musl/8.3.0/../../../../x86_64-linux-musl/bin/as
5953 /usr/bin/mv
3872 /usr/bin/tail
2542 scripts/basic/fixdep
2397 /usr/bin/mkdir
2381 /usr/bin/grep
1926 ./tools/objtool/objtool
1409 /usr/bin/expr
1373 /usr/bin/dirname
1251 /usr/bin/tr
1201 /root/heads/crossgcc/bin/../lib/gcc/x86_64-linux-musl/8.3.0/../../../../x86_64-linux-musl/bin/ld
1200 /root/heads/crossgcc/bin/../libexec/gcc/x86_64-linux-musl/8.3.0/collect2
1176 /usr/bin/make
713 /usr/lib/gcc/x86_64-linux-gnu/9/cc1
666 /usr/bin/as
527 /usr/bin/gcc
512 /root/heads/crossgcc/bin/x86_64-linux-musl-ar
467 /usr/bin/ln
449 /root/heads/build/gawk-4.2.1/gawk
426 /usr/bin/basename
329 /usr/bin/uname
294 /root/heads/crossgcc/bin/i386-linux-musl-gcc
293 /root/heads/crossgcc/bin/../libexec/gcc/i386-linux-musl/8.3.0/cc1
289 /usr/bin/install
286 /root/heads/crossgcc/bin/../lib/gcc/i386-linux-musl/8.3.0/../../../../i386-linux-musl/bin/as
281 /usr/local/bin/make
281 /usr/bin/wc
279 /usr/sbin/make
274 /usr/bin/echo
268 /usr/bin/date
266 /usr/local/sbin/make
257 /usr/bin/tee
253 /root/heads/crossgcc/bin/x86_64-linux-musl-ld
212 /usr/bin/gnatgcc
208 /usr/bin/ld
192 /usr/lib/gcc/x86_64-linux-gnu/9/collect2
187 /usr/bin/cmp
180 /usr/bin/touch
174 /usr/bin/awk
171 /usr/bin/fmt
171 /usr/bin/cp
162 /usr/bin/sort
137 /usr/bin/rmdir
123 /usr/bin/chmod
117 /root/heads/crossgcc/bin/x86_64-linux-musl-strip
103 /usr/bin/hostname
101 /usr/bin/stat
93 /usr/bin/file
89 /usr/bin/which
82 /usr/bin/mktemp
82 /usr/bin/ls
77 /usr/bin/diff
74 /usr/sbin/as
71 /root/heads/crossgcc/bin/x86_64-linux-musl-nm
65 /usr/bin/ar
64 /usr/local/bin/as
63 /usr/local/sbin/as
62 /usr/bin/cut
61 /usr/bin/g++
60 /usr/bin/pkg-config
45 /usr/bin/git
44 /usr/bin/sha256sum
43 /usr/bin/print
39 /usr/bin/tar
36 /usr/bin/ranlib
35 /usr/bin/patch
31 /usr/lib/gcc/x86_64-linux-gnu/9/cc1plus
31 /usr/bin/wget
30 /usr/bin/getconf
30 /root/heads/install/bin/gpg-error-config
30 ./configure
29 /usr/bin/arch
29 /bin/uname
29 /bin/arch
27 /usr/bin/m4
24 /usr/bin/find
23 /usr/bin/sleep
23 /root/heads/crossgcc/bin/i386-linux-musl-ar
22 /usr/bin/uniq
21 /usr/lib/git-core/git
19 /usr/bin/true
18 /usr/bin/dd
18 /root/heads/crossgcc/bin/x86_64-linux-musl-objdump
18 /root/heads/build/slang-2.3.1a/autoconf/mkinsdir.sh
18 /root/heads/build/linux-4.14.62/linux-qemu/tools/objtool//fixdep
17 /usr/bin/bzip2
16 /usr/bin/xargs
15 /usr/sbin/mkdir
15 /usr/local/bin/mkdir
15 /usr/bin/objdump
15 /usr/bin/mt
15 /usr/bin/gawk
15 ./conftest
14 /usr/sbin/file
13 /usr/sbin/which
13 /usr/local/sbin/mkdir
13 /usr/local/bin/rm
13 /usr/local/bin/echo
12 /usr/local/sbin/echo
12 /usr/bin/x86_64-linux-gnu-gcc
12 /usr/bin/gzip
12 /bin/pwd
11 qemu-coreboot/util/cbfstool/cbfstool
11 /usr/sbin/rm
11 /usr/bin/xz
11 /usr/bin/sh
11 /usr/bin/cc
11 /root/heads/crossgcc/bin/i386-linux-musl-objcopy
11 ./gpgrt-config
11 ./gpg-error-config-old
10 /usr/sbin/echo
10 /usr/local/bin/which
9 /usr/local/bin/gcc
9 /usr/bin/bison
9 /root/heads/install/bin/libassuan-config
9 /root/heads/crossgcc/bin/x86_64-linux-musl-readelf
9 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/echo
8 /usr/local/sbin/file
8 /usr/bin/install-info
8 /usr/bin/fgrep
7 /usr/sbin/gcc
7 /usr/local/sbin/rm
7 /usr/local/sbin/gnatgcc
7 /usr/local/sbin/gcc
7 /usr/local/bin/cat
7 /usr/bin/od
7 /bin/machine
6 applets/usage
6 /usr/sbin/gnatgcc
6 /usr/sbin/cp
6 /usr/sbin/cat
6 /usr/local/sbin/which
6 /usr/local/sbin/cat
6 /usr/local/bin/file
6 /usr/bin/nm
6 /usr/bin/head
6 /usr/bin/getopt
6 /root/heads/crossgcc/bin/x86_64-linux-musl-objcopy
6 /root/heads/crossgcc/bin/i386-linux-musl-ld.bfd
6 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/gnatgcc
6 /bin/ln
6 ./util/getrevision.sh
5 /usr/local/bin/gnatgcc
5 /usr/local/bin/cp
5 /usr/bin/flex
5 /usr/bin/cpio
5 /usr/bin/bash
5 /root/heads/install/bin/npth-config
5 /root/heads/install/bin/libgcrypt-config
5 /root/heads/crossgcc/bin/i386-linux-musl-nm
5 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/as
5 ./bin/cpio-clean
4 applets/applet_tables
4 /usr/local/sbin/cp
4 /usr/local/bin/touch
4 /usr/convex/getsysinfo
4 /usr/bin/sha1sum
4 /usr/bin/oslevel
4 /usr/bin/ldd
4 /usr/bin/envsubst
4 /root/heads/install/bin/ksba-config
3 scripts/kconfig/conf
3 qemu-coreboot/util/kconfig/conf
3 /usr/sbin/true
3 /usr/sbin/mv
3 /usr/sbin/expr
3 /usr/local/sbin/sh
3 /usr/local/sbin/bzip2
3 /usr/local/bin/sh
3 /usr/local/bin/mv
3 /usr/local/bin/expr
3 /usr/bin/x86_64-linux-gnu-objdump
3 /usr/bin/x86_64-linux-gnu-as
3 /usr/bin/pwd
3 /usr/bin/hostinfo
3 /usr/bin/autom4te
3 /root/heads/build/busybox-1.28.0/scripts/generate_BUFSIZ.sh
3 /root/heads/build/busybox-1.28.0/scripts/gen_build_files.sh
3 /bin/universe
2 scripts/mod/modpost
2 scripts/kallsyms
2 qemu-coreboot/util/nvramtool/nvramtool
2 autoconf/scripts/getsyslibs.sh
2 arch/x86/entry/vdso/vdso2c
2 applets/usage_compressed
2 /usr/sbin/uname
2 /usr/sbin/sh
2 /usr/sbin/ln
2 /usr/sbin/gzip
2 /usr/sbin/bzip2
2 /usr/local/sbin/true
2 /usr/local/sbin/touch
2 /usr/local/sbin/gzip
2 /usr/local/sbin/getconf
2 /usr/local/sbin/expr
2 /usr/local/sbin/env
2 /usr/local/bin/uname
2 /usr/local/bin/true
2 /usr/local/bin/sha256sum
2 /usr/local/bin/gzip
2 /usr/local/bin/bzip2
2 /usr/lib/git-core/git-submodule
2 /usr/bin/strip
2 /usr/bin/gettext
2 /usr/bin/env
2 /usr/bin/autoconf
2 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/iasl
2 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/gzip
2 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/clang
2 /root/heads/build/busybox-1.28.0/scripts/trylink
2 /bin/false
2 ./scripts/asn1_compiler
2 ./mkheader
2 ./inittests
2 ./a.out
1 util/xcompile/xcompile
1 util/genbuild_h/genbuild_h.sh
1 scripts/mod/mk_elfconfig
1 scripts/conmakehash
1 scripts/basic/split-include
1 qemu-coreboot/util/sconfig/sconfig
1 qemu-coreboot/util/romcc/romcc
1 qemu-coreboot/util/cbfstool/fmaptool
1 lib/gen_crc32table
1 bin/bin-to-hex
1 autoconf/mkinsdir.sh
1 arch/x86/tools/relocs
1 arch/x86/boot/tools/build
1 arch/x86/boot/mkcpustr
1 arch/x86/boot/compressed/mkpiggy
1 applets/usage_pod
1 /usr/sbin/touch
1 /usr/sbin/rmdir
1 /usr/sbin/ldconfig
1 /usr/sbin/install
1 /usr/sbin/grep
1 /usr/sbin/env
1 /usr/sbin/chmod
1 /usr/sbin/bash
1 /usr/local/sbin/strip
1 /usr/local/sbin/sha256sum
1 /usr/local/sbin/ranlib
1 /usr/local/sbin/mv
1 /usr/local/sbin/ln
1 /usr/local/sbin/install
1 /usr/local/sbin/find
1 /usr/local/bin/xz
1 /usr/local/bin/flex
1 /usr/lib/git-core/git-remote-https
1 /usr/bin/pod2text
1 /usr/bin/pod2man
1 /usr/bin/pod2html
1 /usr/bin/egrep
1 /usr/bin/bc
1 /usr/bin/automake
1 /usr/bin/autoheader
1 /sbin/ldconfig.real
1 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/uname
1 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/powerpc64-linux-gnu-clang
1 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin//iasl
1 /root/heads/build/coreboot-4.8.1/qemu-coreboot/cbfstool
1 /root/heads/build/busybox-1.28.0/scripts/mkconfigs
1 /root/heads/bin/cpio-clean
1 ./yat2m
1 ./util/git-hooks/install.sh
1 ./scripts/sortextable
1 ./scripts/basic/bin2c
1 ./runtest
1 ./mkerrcodes
1 ./gpg-error-config-test.sh
1 ./gost-s-box
1 ./autogen.sh
1 ./asn1-gentables
(some of those are spurious -- <unfinished ...>
system calls, for example, are messing up my simplistic grep of the logs to find the successful execve()
calls)
@osresearch I am assuming commands executed within scripts would appear separately. Couldn't shell scripts be removed from this list to simplify it?
Additionally, I am trying to compile Heads and coreboot in Guix. One of the features is the ability to generate deep dependency graphs. I'll post that here as soon as I am able to.
That deep dependency graph is quite neat. I was thinking about making a text version of a dependency tree, although my immediate goal was a little simpler: ensure that we knew which external tools we were using to help identify where reproducibility issues might be creeping in.
For instance, where are the 128 invocations of /usr/bin/as
coming from? Or the 37 /usr/bin/g++
and /usr/bin/ranlib
? On Xen I ran into an issue where older versions of ranlib
would mess up elf headers for some of the special features.
We also have an issue with #892 - /usr/bin/pkg-config
is invoked many times and causes host configuration issues to bleed into the build.
And I'm especially amused by the 10 executions of /usr/bin/mt
; it is probably configure
script, but who has even run a magnetic tape control program in the last decade?
@Thrilleratplay any update, blog post, pointers out to fix zlib problem outlined here and on slack guix-buildstack channel?
@tlaurion Unfortunately, not really. I started down the path of extracting more and more Debian packages to be able to build crossgcc and it got to the point where it was obvious this defeated the purpose of using Guix. Ada released a new version a few weeks ago that may help overcome the bootstrapping issue. I haven't done much with it in the past month; I have been catching up with other obligations in different projects/life. Sorry. I'll try to get back to this soon.
@tlaurion I had sometime last night to take a look at where I was with this. Basically, I keep going around in the same circle trying to resolve dependencies with Ada, Gnat, GCC and Zlib. A few projects have asked or seeked out Ada/Gnat for Guix (example 1 and the Guix IRC logs containing the word gnat). Based on what samplet says here, my problem may be that the AdaCore binaries are not static. This coupled with the fact that the last 32bit AdaCore release was in 2014 and using GCC 4.7.4, this may not be possible ~in a reasonable amount of time~ before the rapture/robot apocalypse. samplet also mentions creating a Ada/Gnat bootstrap using 26 Debian packages and a lot of patchelf. It may have been destined for this repo but it is empty. This was the other path I was heading down but seemed counterproductive to being able to create reproducible environments. This may be the only realistic option until a modern open source Ada crosscompiler is developed. I go back to using Debian binaries.
Interesting enough, #1008 just proved that the issue is more important then known before. On same debian-11 docker image with required tools installed atop (per CircleCI), two builds of the same commit produced different busybox binaries 30 days apart.
This kind of issue will be resolved when a static docker image, without installing additional tools, will be referred per Heads commit ID, pointing CircleCI to use that static docker image. When this is done, a same Heads commit should be able to build the same output (if downloaded tarballs are still accessible of course, which would be #1198 ). And of course, ideally, that docker image would be produced in a reproducible way, which would be #1191 but based on NixOS with a package list pointing to a static (non-live) commit, permitting to always download the same packages and reproducing the same docker image/the same build layer on top of a linux system when dropping under NixOS shell (this is https://nixos.org/guides/towards-reproducibility-pinning-nixpkgs.html)
Long time lurker speaking up. I took a stab at using nix a few weeks ago and ran into ada issues and didn't pursue because it seemed guix was the intended direction. If thats not the case and nix is an option to be considered I'd be happy to give it a stab. If so I'd also go with nix flakes instead of non-flake'd-nix, flakes makes many things easier for projects trying to use nix imo (especially with direnv and devshell too).
@mmlb new to nix myself here. Ada issues will happen if trying to use too new buildstack. Quick test and reports from different users (archlinux issues reported) will pop up when trying to bootstrap older coreboot versions (debian-12 can't build coreboot 4.13 because ada issues) unless patches are applied to coreboot 4.13 (patches/coreboot-4.13 to be applied after coreboot 4.13 tarball decompression)
The easiest would be to bump all boards to coreboot 4.17/4.18 and have newer buildstack bootstrap coreboot musl buildstack.
While continuing to depend on old version of musl-cross-make to build everything not coreboot buildstack.
Any challenge here is more then welcome, of course.
No idea of the differences between nix flakes or nin-flake'd-nix here.
But last time I checked, NixOS had Ada where guix didn't, and as of today, the nixos community and directions seems to make things more easy for Heads to use an OS as a builder more then integrating Heads into derivatives, which was attempted with Guix a while back as a PoC
@mmlb this discussion thread might be helpful to understand the whys and determine the how https://github.com/osresearch/linux-builder/issues/1
@mmlb new to nix myself here. Ada issues will happen if trying to use too new buildstack.
Hmm iirc the issue was with having gcc find the ada compiler. I either had cross-compiling enabled gcc or ada-enabled gcc but the cross gcc did not have ada. Not sure if that was the issue but I'll try to reproduce and get to the bottom of it.
Quick test and reports from different users (archlinux issues reported) will pop up when trying to bootstrap older coreboot versions (debian-12 can't build coreboot 4.13 because ada issues) unless patches are applied to coreboot 4.13 (patches/coreboot-4.13 to be applied after coreboot 4.13 tarball decompression) The easiest would be to bump all boards to coreboot 4.17/4.18 and have newer buildstack bootstrap coreboot musl buildstack.
While continuing to depend on old version of musl-cross-make to build everything not coreboot buildstack.
Yep iirc thats what I was trying to go with too.
Any challenge here is more then welcome, of course. No idea of the differences between nix flakes or nin-flake'd-nix here.
nix flakes is mostly nice here because it has well defined entry points in to the nix setup for the project and lots of tooling is being built up around that.
But last time I checked, NixOS had Ada where guix didn't, and as of today, the nixos community and directions seems to make things more easy for Heads to use an OS as a builder more then integrating Heads into derivatives, which was attempted with Guix a while back as a PoC
Yep most of my use of nix in projects is for a reproducible build environment. Lets aim for that first.
@tlaurion I've started on something would a PR to here be the way to go or into linux-builder?
@mmlb here! but that would be linked to docker #1191 ?
@mmlb : how to use your branch?
Hey @tlaurion idk why I didn't catch these notifications. I'll get a branch and draft PR up here then. It was initially pretty slow going because I kept hitting a gnat bootstrap issue in nixpkgs but I've been able to move past that. My next step was to go rtfm on circle ci to see how I can start testing my branch as the build env for CI, but I'll go ahead and just get the pr up so there's something to look at and maybe get feedback/help with circleci.
A somewhat annoying but workable issue is that a bunch of the external projects that get built have hard coded paths to binaries that don't exist on NixOS or vanilla nix docker image (/bin/pwd
for example) :disappointed: . Things should still work if we're using nix
the tool within another OS but its not as nice and hermetic as I'd like.
@mmlb I quickly checked your commit https://github.com/osresearch/heads/compare/master...mmlb:osresearch-heads:nix, including the patches you added for coreboot 4.13 for gnat that doesn't seem required per https://nixos.wiki/wiki/Coreboot
Here is what I came up with, playing with nixos docker image and nix-shell trying to wrap what i've read in the past to start a PoC myself.
The concept I got with nix-shell is that it permits you to make sure that binaries and libraries available are as defined. First steps here, so learning my way but:
with (import (fetchTarball https://github.com/NixOS/nixpkgs/archive/dfef2e61107dc19c211ead99a5a61374ad8317f4.tar.gz) {}); mkShellNoCC { buildInputs = [ gnat11 # gcc with ada
ncurses # make menuconfig
m4 flex bison # Generate flashmap descriptor parser
#clang
zlib
#acpica-tools # iasl
pkgconfig
gnumake42 # To dodge CircleCI race conditions with make 4.3 and coreboot 4.11, we force make 4.2.1 presence
bashInteractive #TODO: Workaround to have /bin/bash instead of modifying Makefile and other scripts? /usr/bin/env bash?
innoextract #to extract firmware exe
#python39.withPackages([ numpy toolz ]) #WiP. Not sure how this works
python39Full #TODO: Workaround to have /usr/bin/python instead of mydifying python scripts? /usr/bin/env python? Needed for me_cleaner and vbios
unzip #needed by blobs/vbios download and extraction scripts
sudo #needed by bobs/vbios download and extraction scripts #TODO: sudo needs more configuration then just installing. Revisit script and remove...
]; shellHook = ''
NIX_LDFLAGS="$NIX_LDFLAGS -lncurses"
''; }
Of course, I came across discrepencies as commented above. bash python and other paths are searched per shebang in scripts, seem like the better way to do so would be to have those shebang defined as `/usr/bin/env` bash/python which seems portable enough.
Looked into flake a bit as well. Was wondering if necessary and if so why? nix-shell would not be enough?
Steps I see:
- Creating reproducible build environment to build musl-cross-make with dependencies met to use current needed scripts and be able to build coreboot buildstack for multiple coreboot versions as well
- Once successful, investigate NixOS docker image creation to have docker including built musl-cross-make for different archs and currently supported coreboot buildstacks.
- Go from there?
Thoughts?
I have no idea why I'm not seeing the pings from this repo :(, this is the only repo this happens on :fist_raised:...
@mmlb I quickly checked your commit master...mmlb:osresearch-heads:nix, including the patches you added for coreboot 4.13 for gnat that doesn't seem required per nixos.wiki/wiki/Coreboot
Here is what I came up with, playing with nixos docker image and nix-shell trying to wrap what i've read in the past to start a PoC myself.
The concept I got with nix-shell is that it permits you to make sure that binaries and libraries available are as defined. First steps here, so learning my way but:
* pinning package list assures that one cannot deploy newer libraries and tools then available in package list * nix shell doesn't expose nix configuration paths other then what is made available in the shell.
# shell.nix # NOTE we need mkShellNoCC # mkShell would add the regular gcc, which has no ada (gnat) # https://github.com/NixOS/nixpkgs/issues/142943 #tarball pinned to nixos-22.11 as of 12 dec 2022 with (import (fetchTarball https://github.com/NixOS/nixpkgs/archive/dfef2e61107dc19c211ead99a5a61374ad8317f4.tar.gz) {}); mkShellNoCC { buildInputs = [ gnat11 # gcc with ada #gnatboot # gnat1 ncurses # make menuconfig m4 flex bison # Generate flashmap descriptor parser #clang zlib #acpica-tools # iasl pkgconfig gnumake42 # To dodge CircleCI race conditions with make 4.3 and coreboot 4.11, we force make 4.2.1 presence bashInteractive #TODO: Workaround to have /bin/bash instead of modifying Makefile and other scripts? /usr/bin/env bash? innoextract #to extract firmware exe #python39.withPackages([ numpy toolz ]) #WiP. Not sure how this works python39Full #TODO: Workaround to have /usr/bin/python instead of mydifying python scripts? /usr/bin/env python? Needed for me_cleaner and vbios unzip #needed by blobs/vbios download and extraction scripts sudo #needed by bobs/vbios download and extraction scripts #TODO: sudo needs more configuration then just installing. Revisit script and remove... ]; shellHook = '' # TODO remove? NIX_LDFLAGS="$NIX_LDFLAGS -lncurses" ''; }
Of course, I came across discrepencies as commented above. bash python and other paths are searched per shebang in scripts, seem like the better way to do so would be to have those shebang defined as
/usr/bin/env
bash/python which seems portable enough.
Yep this would be fantastic but would take a lot of effort going through all the upstreams to get this fixed. Worthwhile for sure, but not immediate. We could probably make use of buildFHSUserEnv
too though.
Looked into flake a bit as well. Was wondering if necessary and if so why? nix-shell would not be enough?
The big benefit to flakes over nix-shell is that flakes provide known entrypoints and better tooling for known/expected uses. Things like updating nixpkgs pins and better integration with direnv. nix-shell is not necessarily out of the picture but I've enjoyed porting my usage of nix-shell over to flakes. That being said I'm going to try out the shell.nix example you linked to see how it compares with my flake (the mkShellNoCC
may prove to be a big benefit over flake).
Steps I see:
* Creating reproducible build environment to build musl-cross-make with dependencies met to use current needed scripts and be able to build coreboot buildstack for multiple coreboot versions as well * Once successful, investigate NixOS docker image creation to have docker including built musl-cross-make for different archs and currently supported coreboot buildstacks. * Go from there?
Thoughts?
I was wondering if we could bypass musl-cross-make alltogether. It seems silly to use a build tool to build another build tool when the first one could be used alone. But I'd leave that for after we get nix/nixpkgs building the current setup.
Running
strace -f -e execve make
allows us to see what external programs are used and how many times in the build process. If we want to ensure that only trusted/reproducible tools are used, then we might want to build or depend on our own versions of these.