Tune up ONIE build system for Debian 9 (code name stretch)

[cbrune]

One goal of the ONIE project is to make sure ONIE builds correctly on "the currently released version Debian Linux".

The Debian community is going to release Debian 9 soon, so it is time to prepare for it.

This is an overall tracking issue. Additional issue will be created and linked to this issue.

[cbrune]

crosstools NG needs a tune up for compiling gcc-4.7.3 using a host compiler version 6.3.0.

See #486.

[cbrune]

grub2 has problem building.

See #516

[cbrune]

The two outstanding bugs for this umbrella issue are closed.

[bluca]

@cbrune just cloned the repo and tried to follow the suggested make -j4 MACHINE=kvm_x86_64 all recovery-iso on Debian 9, but it fails in so many ways&places. First of all it doesn't look like it's safe to call parallel make - often the build fails, and then running again it progresses. That smells like a race with an undeclared dependency somewhere in the makefiles, so perhaps suggesting -j1 would be safer...

One useful thing I think would be to explicitly list the build dependencies in the README:

build-essential stgit u-boot-tools util-linux \
                                  gperf device-tree-compiler python-all-dev xorriso \
                                  autoconf automake bison flex texinfo libtool libtool-bin \
                                  realpath gawk libncurses5 libncurses5-dev bc \
                                  dosfstools mtools pkg-config git wget help2man libexpat1 \
                                  libexpat1-dev fakeroot python-sphinx rst2pdf \
                                  libefivar-dev libnss3-tools libnss3-dev libpopt-dev \
                                  libssl-dev efitools uuid-runtime

As I for one am not in the habit of downloading random software and running it as root :-)

More importantly, I had to export CPPFLAGS="-P" to get past an ncurses build error: https://stackoverflow.com/questions/37475222/ncurses-6-0-compilation-error-error-expected-before-int

Then, I had to downgrade libssl-dev to libssl1.0-dev as the version of sbsigntool you ship does not support openssl 1.1.x - you might want to get the patches that are shipped in the Debian sbsigntool package that adds compatibility with the new API: https://tracker.debian.org/pkg/sbsigntool - not sure if backward compat is maintained though - also beware that the kernel +sbsigntool + pesign MUST be built with the same openssl version otherwise the kbuild sign program will barf when signing the modules for EFI Secure Boot. Been there...

Then, right now I'm stuck trying to build dropbear:

ccache gcc --sysroot=/home/bluca/git/onie/build/user/x86_64-g6.3.0-lnx4.9.80-uClibc-ng-1.0.22/dev-sysroot -o dropbearmulti dbmulti.o atomicio.o bignum.o buffer.o circbuffer.o cli-agentfwd.o cli-auth.o cli-authinteract.o cli-authpasswd.o cli-authpubkey.o cli-channel.o cli-chansession.o cli-kex.o cli-main.o cli-runopts.o cli-session.o cli-tcpfwd.o common-algo.o common-channel.o common-chansession.o common-kex.o common-runopts.o common-session.o compat.o crypto_desc.o curve25519-donna.o dbhelpers.o dbrandom.o dbutil.o dh_groups.o dropbearconvert.o dropbearkey.o dss.o ecc.o ecdsa.o fake-rfc2553.o gendss.o genrsa.o gensignkey.o keyimport.o list.o listener.o loginrec.o ltc_prng.o netio.o packet.o process-packet.o progressmeter.o queue.o rsa.o scp.o scpmisc.o signkey.o sshpty.o svr-agentfwd.o svr-auth.o svr-authpam.o svr-authpasswd.o svr-authpubkey.o svr-authpubkeyoptions.o svr-chansession.o svr-kex.o svr-main.o svr-runopts.o svr-service.o svr-session.o svr-tcpfwd.o svr-x11fwd.o tcp-accept.o termcodes.o libtomcrypt/libtomcrypt.a libtommath/libtommath.a  -lutil -lz  -lcrypt
cli-kex.o: In function `ask_to_confirm':
cli-kex.c:(.text+0xa6): undefined reference to `__GETC_UNLOCKED_MACRO'
cli-kex.c:(.text+0xe9): undefined reference to `__GETC_UNLOCKED_MACRO'
cli-runopts.o: In function `match_extendedopt':
cli-runopts.c:(.text+0x58): undefined reference to `__ctype_b'
dbutil.o: In function `buf_getline':
dbutil.c:(.text+0x2a6): undefined reference to `__fgetc_unlocked'
keyimport.o: In function `load_openssh_key':
keyimport.c:(.text+0x289): undefined reference to `__ctype_b'
keyimport.o: In function `openssh_write':
keyimport.c:(.text+0x1032): undefined reference to `__fputc_unlocked'
scp.o: In function `okname':
scp.c:(.text+0x8f2): undefined reference to `__ctype_b'
scp.o: In function `sink':
scp.c:(.text+0x16ac): undefined reference to `__ctype_b'
scpmisc.o: In function `fatal':
scpmisc.c:(.text+0x2b9): undefined reference to `__fputc_unlocked'
libtommath/libtommath.a(bn_mp_read_radix.o): In function `mp_read_radix':
bn_mp_read_radix.c:(.text+0x5e): undefined reference to `__ctype_toupper'
collect2: error: ld returned 1 exit status
Makefile:187: recipe for target 'dropbearmulti' failed

Still haven't got past this one.

[bluca]

Tried latest master, 2017.11 and 2018.02 and that error is always there. Any idea?

[bluca]

Tried also with gcc-4.8 and gcc-4.9, same errors. They happen with various programs, eg:

gcc-4.9 -Os --sysroot=/home/bluca/git/onie/build/user/x86_64-g6.3.0-lnx4.9.40-uClibc-ng-1.0.22/dev-sysroot  -fPIC  -O2 --sysroot=/home/bluca/git/onie/build/user/x86_64-g6.3.0-lnx4.9.40-uClibc-ng-1.0.22/dev-sysroot -O2 -L../libdm -L../lib -L../libdaemon/client -Wl,-z,relro,-z,now -pie -L../libdm \
      -o dmsetup dmsetup.o -ldevmapper   
/usr/bin/ld: warning: libpthread.so.0, needed by ../libdm/libdevmapper.so, not found (try using -rpath or -rpath-link)
dmsetup.o: In function `_out_char':
dmsetup.c:(.text+0xea5): undefined reference to `__stdout'
dmsetup.c:(.text+0xeed): undefined reference to `__stdout'
dmsetup.c:(.text+0xf33): undefined reference to `__stdout'
dmsetup.c:(.text+0x1013): undefined reference to `__fputc_unlocked'
dmsetup.c:(.text+0x1076): undefined reference to `__fputc_unlocked'
dmsetup.c:(.text+0x1086): undefined reference to `__fputc_unlocked'
dmsetup.o: In function `_parse_line':
dmsetup.c:(.text+0x1160): undefined reference to `__ctype_b'
dmsetup.c:(.text+0x119d): undefined reference to `__ctype_b'
dmsetup.o: In function `_display_tree_node.isra.11':
dmsetup.c:(.text+0x2b58): undefined reference to `__stdout'
dmsetup.c:(.text+0x2d47): undefined reference to `__stdout'
dmsetup.c:(.text+0x2e54): undefined reference to `__fputc_unlocked'
dmsetup.c:(.text+0x2ec7): undefined reference to `__fputc_unlocked'
dmsetup.o: In function `_status':
dmsetup.c:(.text+0x44fc): undefined reference to `__ctype_b'
dmsetup.o: In function `_out_char':
dmsetup.c:(.text+0x1007): undefined reference to `__fputc_unlocked'
dmsetup.c:(.text+0x105c): undefined reference to `__fputc_unlocked'
dmsetup.o: In function `main':
dmsetup.c:(.text.startup+0x16a9): undefined reference to `__ctype_b'
../libdm/libdevmapper.so: undefined reference to `__ctype_toupper'
collect2: error: ld returned 1 exit status
Makefile:127: recipe for target 'dmsetup' failed

Probably it's picking the wrong libc. Sounds like the Makefiles that define the build environment are a bit broken.

[bluca]

Gah - it was ccache.

Given it breaks everything, please consider adding a check in the Makefile, and if CC contains ccache throw an error. Would have saved me a couple of hours of head scratching...

[cbrune]

Hello @bluca -

Sorry to hear to hit some snags.

Admittedly, the initial build dependencies for the crosstools environment is not perfect. The number one problem new folks encounter is that during the build of the crosstools, a build host dependency is missing and the build stops.

Next the user installs the missing package(s) and tryies to start where they left off with make -j4 MACHINE=kvm_x86_64 all recovery-iso. Trying to re-start the crosstool build, which is the first thing the ONIE build system needs to build, does not work. That is a bug, just haven't prioritized very high.

This is, however, a one time problem, as for subsequent attempts all the build host dependency packages are installed.

To move forward, I recommend running make distclean whenever you have to install a build dependency. Once installing all the build dependencies is complete, run make distclean. After that make -j4 MACHINE=kvm_x86_64 all works fine.

After the one time build host setup, parallel make works great. Restarting it works fine also -- except in the case of building the crosstools, which you have hit.

I just checked out 2017.11 on a fresh Debian 9 VM and built the kvm_x86_64 machine without problems. I installed the build dependencies first.

One useful thing I think would be to explicitly list the build dependencies in the README: As I for one am not in the habit of downloading random software and running it as root :-)

That is a good suggestion to list the dependencies explicitly. There is some documentation already here: https://opencomputeproject.github.io/onie/developers/building.html#preparing-a-new-build-machine

The Makefile target uses sudo, so it is not explicitly asking people to run anything as root. The user has the choice not to run it. Also they are free to view the Makefile.

If you are still having problems after trying make distclean that would be interesting.

[cbrune]

Hello @bluca -- glad to hear to you reasoned out the problem.

[bluca]

I would suggest to add a check for ccache - it's a very widely used tool, so I'm not the only one that might be tripped by it.

The problem with sbsigntool and libssl-dev is also still there in newer versions.

[cbrune]

Yes, there is a problem in master with sbsigntool which is WIP.

[bluca]

@cbrune FYI: I've updated live-build and live-boot in Debian (currently in unstable/sid, will be available in testing/buster in a few days) to be able to build native ONIE-compatible images that can be booted with onie-nos-install.

https://tracker.debian.org/news/943545/accepted-live-boot-120180328-source-into-unstable/ https://tracker.debian.org/news/943547/accepted-live-build-120180328-source-into-unstable/

Whoever uses live-build can simply add --onie true (and optionally --onie-kernel-cmdline for additional parameters to pass when booting) to the lb config command when building live images and they'll get a live-image-name.hybrid.iso-ONIE.bin file.

The code is in this file: https://salsa.debian.org/live-team/live-build/blob/master/scripts/build/binary_onie and was inspired by the contrib/debian-iso/sharch_body.sh and contrib/debian-iso/cook-bits.sh scripts in this repo. (NB: most of the work was done by my colleague Erik)

[cbrune]

@bluca -- this is totally awesome!! I am blown a way. Soooo cool!