error: err changed binding to STB_GLOBAL

[tariqkurd-repo]

I'm running on linux, ubuntu18.04

I ran this:

./cheribuild.py --source-root ./download llvm-native

But had to hack pycheribuild/projects/project.py so that the cmake command line was correct (it was putting -j8 before the -- so it wasn't passed to make, and cmake said no such argument), so I have:

    # For CMake all other options are now forwarded to the actual tool
    if self.kind == MakeCommandKind.CMake:
        assert self.subkind is not None
        result.append("--")
    if jobs and self.can_pass_jflag:  #put after the -- not before
        result.append("-j" + str(jobs))

and then llvm-native built correctly. I moved onto llvm-riscv64:

./cheribuild.py --source-root ./download llvm-riscv64 -d

I get this error when I build:

===> tools/build (obj,includes,all,install) error: err changed binding to STB_GLOBAL 1 error generated. --- err.o --- [err.o] Error code 1 bmake[3]: stopped in /data/toolchains/riscv/CHERI/cheribuild/download/cheribsd/tools/build 1 error bmake[3]: stopped in /data/toolchains/riscv/CHERI/cheribuild/download/cheribsd/tools/build --- legacy --- [legacy] Error code 2 bmake[2]: stopped in /data/toolchains/riscv/CHERI/cheribuild/download/cheribsd 1 error bmake[2]: stopped in /data/toolchains/riscv/CHERI/cheribuild/download/cheribsd --- _legacy --- --- buildworld --- Fatal error: Command nice /data/toolchains/riscv/CHERI/cheribuild/download/cheribsd/tools/build/make.py buildworld -j8 -DDB_FROM_SRC -DI_REALLY_MEAN_NO_CLEAN -DNO_ROOT -DBUILD_WITH_STRICT_TMPPATH TARGET_ARCH=riscv64 TARGET=riscv DEBUG_FLAGS=-g 'LOCAL_XTOOL_DIRS=lib/libnetbsd usr.sbin/makefs usr.bin/mkimg' XAR=/data/toolchains/riscv/CHERI/cheribuild/download/output/sdk/bin/llvm-ar XNM=/data/toolchains/riscv/CHERI/cheribuild/download/output/sdk/bin/llvm-nm XSIZE=/data/toolchains/riscv/CHERI/cheribuild/download/output/sdk/bin/llvm-size XSTRIPBIN=/data/toolchains/riscv/CHERI/cheribuild/download/output/sdk/bin/llvm-strip XSTRINGS=/data/toolchains/riscv/CHERI/cheribuild/download/output/sdk/bin/llvm-strings XOBJCOPY=/data/toolchains/riscv/CHERI/cheribuild/download/output/sdk/bin/llvm-objcopy XRANLIB=/data/toolchains/riscv/CHERI/cheribuild/download/output/sdk/bin/llvm-ranlib -DWITHOUT_CLEAN -DWITH_TESTS -DWITHOUT_INIT_ALL_ZERO -DWITHOUT_INIT_ALL_PATTERN -DWITHOUT_MAN -DWITHOUT_MAIL -DWITHOUT_DEBUG_FILES -DWITHOUT_PROFILE -DWITHOUT_OFED -DWITH_GOOGLETEST -DWITHOUT_GCC -DWITHOUT_CLANG -DWITHOUT_LLD -DWITHOUT_LLDB -DWITHOUT_GCC_BOOTSTRAP -DWITHOUT_CLANG_BOOTSTRAP -DWITHOUT_LLD_BOOTSTRAP -DWITHOUT_LIB32 -DWITH_ELFTOOLCHAIN_BOOTSTRAP -DWITH_TOOLCHAIN -DWITHOUT_BINUTILS_BOOTSTRAP -s -de failed with non-zero exit code 2

I'm happy to report tool versions / environment variables if you can tell which which commands to run to report them.

Tariq

[jrtc27]

Several questions/points:

1. What is your system toolchain? It sounds like you have an LLVM 12 being used? Although we've fixed cross-building with LLVM 12, it's quite possible there's something in tools/build for bootstrapping a native build environment on Linux+macOS that breaks if built with LLVM 12.
2. I've never seen that issue with CMake, though we've since had to move to newer CMake versions as post-LLVM 11 it requires a version newer than 18.04 provides. I suspect newer versions support -jN before the -- but older ones don't. Could you please try with an updated CMake? You should be able to build one with cheribuild, though I haven't used it myself.
3. You're building llvm-riscv64, i.e. an LLVM compiler that will run on a plain RISC-V CheriBSD (which is basically FreeBSD at that point) system, hence why it's first building CheriBSD so it has a sysroot with headers and libraries to use when cross-building LLVM. I doubt that's what you want?

[tariqkurd-repo]

Hi @jrtc27 ,

LLVM6? Sounds a bit old but reports to be the newest available:

tariq@lhrptariq00002:/data/toolchains/riscv/CHERI/cheribuild$ /usr/bin/clang --version clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

tariq@lhrptariq00002:/data/toolchains/riscv/CHERI/cheribuild$ sudo apt upgrade clang Reading package lists... Done Building dependency tree
Reading state information... Done clang is already the newest version (1:6.0-41~exp5~ubuntu1). Calculating upgrade... Done

cmake claims to be the latest (probably because of the older ubuntu version)

tariq@lhrptariq00002:/data/toolchains/riscv/CHERI/cheribuild$ sudo apt upgrade cmake Reading package lists... Done Building dependency tree
Reading state information... Done cmake is already the newest version (3.10.2-1ubuntu2.18.04.1). Calculating upgrade... Done

In terms of the target toolchain, I would like to cross-compile CHERI code, running on x86 ubuntu and compiling for RISCV.

What's the correct target to build for that?

Thanks

Tariq

[jrtc27]

Hi @jrtc27 ,

LLVM6? Sounds a bit old but reports to be the newest available:

tariq@lhrptariq00002:/data/toolchains/riscv/CHERI/cheribuild$ /usr/bin/clang --version clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

tariq@lhrptariq00002:/data/toolchains/riscv/CHERI/cheribuild$ sudo apt upgrade clang Reading package lists... Done Building dependency tree Reading state information... Done clang is already the newest version (1:6.0-41~exp5~ubuntu1). Calculating upgrade... Done

Is that what cheribuild is using though? Please post the start of the log where cheribuild prints out what toolchain its using and the exact environment it's using. It's possible building with Clang 6 is indeed broken though, it's ancient, but I thought we supported 6 fine... That's just the default clang version though, there is a clang-8 you can install that might fare better

cmake claims to be the latest (probably because of the older ubuntu version)

tariq@lhrptariq00002:/data/toolchains/riscv/CHERI/cheribuild$ sudo apt upgrade cmake Reading package lists... Done Building dependency tree Reading state information... Done cmake is already the newest version (3.10.2-1ubuntu2.18.04.1). Calculating upgrade... Done

Yes, that's why I mentioned building one with cheribuild. You'll need to either do that or build it from source yourself manually (it's easy to build though).

In terms of the target toolchain, I would like to cross-compile CHERI code, running on x86 ubuntu and compiling for RISCV.

What's the correct target to build for that?

Then you're wrong for a couple of reasons; firstly, llvm-riscv64 is building a compiler that runs on RISC-V (and what it builds for isn't relevant, LLVM isn't like GCC, every LLVM is a cross-compiler out of the box for all architectures unless you configure it to have fewer backends), and secondly riscv64 means plain 64-bit RISC-V not CHERI-RISC-V whenever you see it in cheribuild targets.

As for what you want, you want cheribsd-sdk-riscv64-purecap.

Oh and one final point: I don't think we've ever intended for --source-root=relative/path to be supported. In practice it is because we never change the working directory of the process, only children that have had everything pre-computed, but I wouldn't be surprised if there are things that break if you do that. Please just give it an absolute path (and you probably want to put it in your ~/.config/cheribuild.json so you don't need to repeat it all the time).

[tariqkurd-repo]

Hi again,

OK I'm now building:

./cheribuild.py --source-root $PWD/download cheribsd-sdk-riscv64-purecap

It says:

Checking if /usr/bin/clang-12 can be used to build FreeBSD... /usr/lib/llvm-12/bin/clang (clang version 12.0.1) can be used to build FreeBSD.

But then it says:

ERROR: Your compiler does not support the __thread specifier for Thread-Local Storage (TLS). Please upgrade to a version that does.

Fatal error: Command /data/toolchains/riscv/CHERI/cheribuild/download/qemu/configure --disable-sdl --disable-gtk --disable-opengl --python=/usr/bin/python3 --disable-stack-protector --disable-strip --smbd=/usr/sbin/smbd --target-list=cheri128-softmmu,mips64-softmmu,riscv64-softmmu,riscv64cheri-softmmu,riscv32-softmmu,x86_64-softmmu,aarch64-softmmu --enable-slirp=git --disable-linux-user --disable-bsd-user --disable-xen --disable-docs --disable-rdma --disable-werror --disable-pie '--extra-cflags=-flto -O3 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -Werror=return-type -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -Werror=incompatible-pointer-types-discards-qualifiers -Wno-address-of-packed-member -Wextra -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers' --cxx=/usr/bin/clang++-12 --cc=/usr/bin/clang-12 --make=make '--extra-ldflags=-fuse-ld=/usr/lib/llvm-12/bin/ld.lld -flto' '--extra-cxxflags=-flto -O3 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -Werror=return-type' --disable-capstone --prefix=/data/toolchains/riscv/CHERI/cheribuild/download/output/sdk failed with non-zero exit code 1

Any idea how to proceed with this build?

Thanks

Tariq

[jrtc27]

Hi again,

OK I'm now building:

./cheribuild.py --source-root $PWD/download cheribsd-sdk-riscv64-purecap

It says:

Checking if /usr/bin/clang-12 can be used to build FreeBSD... /usr/lib/llvm-12/bin/clang (clang version 12.0.1) can be used to build FreeBSD.

That explains why you get errors that look like ones introduced in LLVM 12 when building CheriBSD, though I'm not sure why those are being generated. Could you see if using --cheribsd/git-revision=dev makes things any better? It's possible there's an LLVM 12 fix in there for cross-building that also fixes bootstrapping with LLVM 12 (our LLVM master is still 11, so CheriBSD master doesn't claim to support LLVM 12 yet).

But then it says:

ERROR: Your compiler does not support the __thread specifier for Thread-Local Storage (TLS). Please upgrade to a version that does.

Fatal error: Command /data/toolchains/riscv/CHERI/cheribuild/download/qemu/configure --disable-sdl --disable-gtk --disable-opengl --python=/usr/bin/python3 --disable-stack-protector --disable-strip --smbd=/usr/sbin/smbd --target-list=cheri128-softmmu,mips64-softmmu,riscv64-softmmu,riscv64cheri-softmmu,riscv32-softmmu,x86_64-softmmu,aarch64-softmmu --enable-slirp=git --disable-linux-user --disable-bsd-user --disable-xen --disable-docs --disable-rdma --disable-werror --disable-pie '--extra-cflags=-flto -O3 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -Werror=return-type -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -Werror=incompatible-pointer-types-discards-qualifiers -Wno-address-of-packed-member -Wextra -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers' --cxx=/usr/bin/clang++-12 --cc=/usr/bin/clang-12 --make=make '--extra-ldflags=-fuse-ld=/usr/lib/llvm-12/bin/ld.lld -flto' '--extra-cxxflags=-flto -O3 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -Werror=return-type' --disable-capstone --prefix=/data/toolchains/riscv/CHERI/cheribuild/download/output/sdk failed with non-zero exit code 1

Any idea how to proceed with this build?

https://github.com/CTSRD-CHERI/qemu/blob/qemu-cheri/configure#L2111-L2120 is the check that fails, unmodified from upstream QEMU. As you can see it's rather simple. Please look at ~/cheri/build/qemu-build/config.log to find the error that caused (or upload it if it's not obvious to you); that really should not be failing.

[tariqkurd-repo]

Thanks - that's really helpful.

config.log says there's an error when compiling

/usr/bin/clang-12 -std=gnu99 -Wall -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -flto -O3 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -Werror=return-type -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -Werror=incompatible-pointer-types-discards-qualifiers -Wno-address-of-packed-member -Wextra -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c config-temp/qemu-conf.c:2:2: error: i386 not defined #error i386 not defined ^

but if I run the same command on the command line it's fine, so there's something going wrong with the build environment but I don't know what.

Tariq

[jrtc27]

There will be tons of errors as it tries to discover various properties of the host, that's to be expected. You likely want to see what the last error in the file is.

[tariqkurd-repo]

/usr/bin/clang-12 -std=gnu99 -Wall -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -flto -O3 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -Werror=return-type -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -Werror=incompatible-pointer-types-discards-qualifiers -Wno-address-of-packed-member -Wextra -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Werror -Wformat-y2k -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -flto -O3 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -Werror=return-type -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -Werror=incompatible-pointer-types-discards-qualifiers -Wno-address-of-packed-member -Wextra -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -fuse-ld=/usr/lib/llvm-12/bin/ld.lld -flto

clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path]

it's strange - some files generate this as a warning, but some as an error.

Some of them have -Werror specified, but 2 of them don't.

grep -i error config.log | grep -v 'Werror='

returns

config-temp/qemu-conf.c:2:2: error: i386 not defined

error i386 not defined

1 error generated. config-temp/qemu-conf.c:2:2: error: ILP32 not defined

error ILP32 not defined

1 error generated. clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path] clang: error: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Werror,-Wfuse-ld-path]

[jrtc27]

Ok, that makes sense, I see the problem (as for why it's only sometimes an error, some of the configure checks pass -Werror, some don't).

[tariqkurd-repo]

Yeah - any idea how to progress? If it has been a warning in all cases many of the checks would have passed. Should I hassle QEMU folk directly about this, and what would be the right way to do that? Thanks for your help @jrtc27 ! Maybe I can try hacking the script so it's always a warning.....

[jrtc27]

The QEMU issue should be resolved by https://github.com/CTSRD-CHERI/cheribuild/commit/1359bfe96c1e5f036ba453b3d6ce1d2e00c1dd0a.

[jrtc27]

I don't believe there are any remaining issues here