systemd-nspawn: undefined symbol: strjoin

[zeysh]

Hi,

On my fresh Ubuntu install, I got this issue with dgr:

sudo dgr build                                                          
18:15:31 INFO            dgr/dgr/aci-build.go:52  Building                                     aci=blablacar.github.io/dgr/aci-base
18:15:32 INFO            dgr/dgr/aci-build.go:219 Importing build to rkt                       aci=blablacar.github.io/dgr/aci-base path=/home/***/github/blablacar/dgr/examples/aci-base/target/builder/image.aci
18:15:33 INFO            dgr/dgr/aci-build.go:72  Calling rkt to start build                   aci=blablacar.github.io/dgr/aci-base
18:15:33 INFO          d/a/bin-run/builder.go:66  Building aci                                 aci=aci-base
/var/lib/rkt/pods/run/1891be75-86fe-45bf-b630-acfd6b29b6ac/stage1/rootfs/dgr/usr/bin/systemd-nspawn: symbol lookup error: /var/lib/rkt/pods/run/1891be75-86fe-45bf-b630-acfd6b29b6ac/stage1/rootfs/dgr/usr/bin/systemd-nspawn: undefined symbol: strjoin
18:15:33 FATAL  d/aci-builder/bin-run/main.go:27  Build failed                                
                       d/a/bin-run/builder.go:198 Builder run failed                           aci=aci-base
                                                  exit status 127
18:15:33 FATAL             dgr/dgr/command.go:166 Build command failed                        
                         dgr/dgr/aci-build.go:75  Builder container return with failed status  aci=blablacar.github.io/dgr/aci-base
                 dgr/dgr/common/rkt-client.go:238 Run failed                                   config={Path: InsecureOptions:[ondisk image] dir: LocalConfig: SystemConfig: UserConfig: TrustKeysFromHttps:false NoStore:false StoreOnly:false}
                                                  exit status 1

Here my setup:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 17.04
Release:    17.04
Codename:   zesty

sudo dgr --version
dgr

version    : 85
build date : 2017-06-28_09:53:52_UTC
CommitHash : 71babe9

$sudo rkt version
rkt Version: 1.27.0
appc Version: 0.8.10
Go Version: go1.7.4
Go OS/Arch: linux/amd64
Features: -TPM +SDJOURNAL

I guess I missed something... Let me know if you need more informations.

Many thanks

[n0rad]

Looks like an incompatiblity between host libs and builder's stage1 libs, that actually should not exists.

Long story: Builder in the stage1 start systemd-nspawn to isolate the build using a nspawn binary in the stage1. nspawn is started using the linker directly to use only lib from the builder : usr/lib/ld-linux-x86-64.so.2 usr/bin/systemd-nspawn ... and LD_LIBRARY_PATH is set as env variable to builder lib dir

but

LD_LIBRARY_PATH fallback to host libs if some are missing and the stage1 is missing some libs for nspawn:

# ldd usr/bin/systemd-nspawn
        linux-vdso.so.1 =>  (0x00007ffc33d7c000)
        libsystemd-shared-232.so => /lib/systemd/libsystemd-shared-232.so (0x00007efe39983000)
        libblkid.so.1 => /var/lib/rkt/pods/run/ad8fe634-fa7f-4c35-b94e-b08b6eb6c406/stage1/rootfs/dgr/lib64/libblkid.so.1 (0x00007efe39748000)
        libpthread.so.0 => /var/lib/rkt/pods/run/ad8fe634-fa7f-4c35-b94e-b08b6eb6c406/stage1/rootfs/dgr/lib64/libpthread.so.0 (0x00007efe3952b000)
        libc.so.6 => /var/lib/rkt/pods/run/ad8fe634-fa7f-4c35-b94e-b08b6eb6c406/stage1/rootfs/dgr/lib64/libc.so.6 (0x00007efe3918d000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007efe38f5f000)
        libcap.so.2 => /var/lib/rkt/pods/run/ad8fe634-fa7f-4c35-b94e-b08b6eb6c406/stage1/rootfs/dgr/lib64/libcap.so.2 (0x00007efe38d5a000)
        librt.so.1 => /var/lib/rkt/pods/run/ad8fe634-fa7f-4c35-b94e-b08b6eb6c406/stage1/rootfs/dgr/lib64/librt.so.1 (0x00007efe38b52000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007efe3892c000)
        liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 (0x00007efe38714000)
        libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007efe38405000)
        libacl.so.1 => /lib/x86_64-linux-gnu/libacl.so.1 (0x00007efe381fb000)
        libidn.so.11 => /lib/x86_64-linux-gnu/libidn.so.11 (0x00007efe37fc8000)
        libseccomp.so.2 => /lib/x86_64-linux-gnu/libseccomp.so.2 (0x00007efe37d83000)
        /lib64/ld-linux-x86-64.so.2 (0x000055796b310000)
        libuuid.so.1 => /var/lib/rkt/pods/run/ad8fe634-fa7f-4c35-b94e-b08b6eb6c406/stage1/rootfs/dgr/lib64/libuuid.so.1 (0x00007efe37b7f000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007efe3790c000)
        libdl.so.2 => /var/lib/rkt/pods/run/ad8fe634-fa7f-4c35-b94e-b08b6eb6c406/stage1/rootfs/dgr/lib64/libdl.so.2 (0x00007efe37706000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007efe374f2000)
        libattr.so.1 => /var/lib/rkt/pods/run/ad8fe634-fa7f-4c35-b94e-b08b6eb6c406/stage1/rootfs/dgr/lib64/libattr.so.1 (0x00007efe372ed000)

adding --library-path usr/lib64 to the linker looks to enforce using only stage1 libs and fail on missing first lib (seccomp).

to fix:

• Add missing libs to builder
• add --library-path usr/lib64 to prevent fallback on host libs ?

[n0rad]

With deep look, it's only libsystemd-shared-232.so that is loaded from host on debian, whatever library path is set.

I forced loading the one from stage1 using ld_preload.

[n0rad]

fixed with v86