undefined reference to `_impure_ptr' when building Zephyr C++ samples on non-Linux hosts

[stephanosio]

Zephyr C++ samples and tests fail to link with "undefined reference to `_impure_ptr'" on macOS and Windows hosts using the respective host toolchains.

For example, building samples/cpp/hello_world for qemu_arc_hs on a Windows host results in the following failure:

[99/104] Linking CXX executable zephyr\zephyr_pre0.elf

FAILED: zephyr/zephyr_pre0.elf zephyr/zephyr_pre0.map C:/a/sdk-ng/sdk-ng/test/twister-out/twister_links/test_442/zephyr/zephyr_pre0.map 

C:\Windows\system32\cmd.exe /C "cd . && ccache S:\arc-zephyr-elf\bin\arc-zephyr-elf-g++.exe  -gdwarf-4 zephyr/CMakeFiles/zephyr_pre0.dir/misc/empty_file.c.obj -o zephyr\zephyr_pre0.elf  zephyr/CMakeFiles/offsets.dir/./arch/arc/core/offsets/offsets.c.obj  -fuse-ld=bfd  -T  zephyr/linker_zephyr_pre0.cmd  -Wl,-Map=C:/a/sdk-ng/sdk-ng/test/twister-out/twister_links/test_442/zephyr/zephyr_pre0.map  -Wl,--whole-archive  app/libapp.a  zephyr/libzephyr.a  zephyr/arch/common/libarch__common.a  zephyr/arch/arch/arc/core/libarch__arc__core.a  zephyr/arch/arch/arc/core/mpu/libarch__arc__core__mpu.a  zephyr/lib/libc/picolibc/liblib__libc__picolibc.a  zephyr/lib/libc/common/liblib__libc__common.a  zephyr/drivers/interrupt_controller/libdrivers__interrupt_controller.a  zephyr/drivers/console/libdrivers__console.a  zephyr/drivers/serial/libdrivers__serial.a  zephyr/drivers/timer/libdrivers__timer.a  -Wl,--no-whole-archive  zephyr/kernel/libkernel.a  -L"s:/arc-zephyr-elf/bin/../lib/gcc/arc-zephyr-elf/12.2.0/archs"  -LC:/a/sdk-ng/sdk-ng/test/twister-out/twister_links/test_442/zephyr  -lgcc  zephyr/arch/common/libisr_tables.a  -Wl,--fatal-warnings  -mcpu=archs  -Wl,--gc-sections  -Wl,--build-id=none  -Wl,--sort-common=descending  -Wl,--sort-section=alignment  -Wl,-u,_OffsetAbsSyms  -Wl,-u,_ConfigAbsSyms  -nostdlib  -static  -Wl,-X  -Wl,-N  -Wl,--orphan-handling=warn  -lstdc++  -Wl,-no-pie  -DPICOLIBC_LONG_LONG_PRINTF_SCANF  --specs=picolibc.specs  -lc  -lgcc && C:\Windows\system32\cmd.exe /C "cd /D C:\a\sdk-ng\sdk-ng\test\twister-out\twister_links\test_442\zephyr && "C:\Program Files\CMake\bin\cmake.exe" -E true""
s:/arc-zephyr-elf/bin/../lib/gcc/arc-zephyr-elf/12.2.0/../../../../arc-zephyr-elf/bin/ld.bfd.exe: s:\arc-zephyr-elf\bin\../lib/gcc/../../picolibc/arc-zephyr-elf/lib/archs\libstdc++.a(ios_init.o): in function `std::ios_base::Init::Init()':

ios_init.cc:(.text._ZNSt8ios_base4InitC2Ev+0x1c): undefined reference to `_impure_ptr'

s:/arc-zephyr-elf/bin/../lib/gcc/arc-zephyr-elf/12.2.0/../../../../arc-zephyr-elf/bin/ld.bfd.exe: ios_init.cc:(.text._ZNSt8ios_base4InitC2Ev+0x1c): undefined reference to `_impure_ptr'

s:/arc-zephyr-elf/bin/../lib/gcc/arc-zephyr-elf/12.2.0/../../../../arc-zephyr-elf/bin/ld.bfd.exe: ios_init.cc:(.text._ZNSt8ios_base4InitC2Ev+0x3e): undefined reference to `_impure_ptr'

s:/arc-zephyr-elf/bin/../lib/gcc/arc-zephyr-elf/12.2.0/../../../../arc-zephyr-elf/bin/ld.bfd.exe: ios_init.cc:(.text._ZNSt8ios_base4InitC2Ev+0x3e): undefined reference to `_impure_ptr'

s:/arc-zephyr-elf/bin/../lib/gcc/arc-zephyr-elf/12.2.0/../../../../arc-zephyr-elf/bin/ld.bfd.exe: ios_init.cc:(.text._ZNSt8ios_base4InitC2Ev+0x5e): undefined reference to `_impure_ptr'

s:/arc-zephyr-elf/bin/../lib/gcc/arc-zephyr-elf/12.2.0/../../../../arc-zephyr-elf/bin/ld.bfd.exe: s:\arc-zephyr-elf\bin\../lib/gcc/../../picolibc/arc-zephyr-elf/lib/archs\libstdc++.a(ios_init.o):ios_init.cc:(.text._ZNSt8ios_base4InitC2Ev+0x5e): more undefined references to `_impure_ptr' follow

collect2.exe: error: ld returned 1 exit status

ninja: build stopped: subcommand failed.

Building the same sample for the same target on a Linux host using the Linux host toolchain does not result in the above failure; suggesting the libstdc++ (or maybe picolibc?) build is somehow done differently for non-Linux host toolchain builds.

[moritz-geier]

Does that mean that it is currently not possible to compile C++ Code on Windows?

[keith-packard]

It "should" work as the toolchains are supposed to be the same. But you're pretty clearly picking up some newlib bits somehow?

[Finomnis]

I can confirm that this fails for me as well on windows, using zephyr 3.6.0.

[keith-packard]

Let's check how ios_init.cc gets compiled -- somehow it's pulling in a newlib header instead of the matching picolibc header.

[stephanosio]

Let's check how ios_init.cc gets compiled -- somehow it's pulling in a newlib header instead of the matching picolibc header.

@keith-packard By any chance, have you had a chance to figure out what is wrong?

[keith-packard]

@keith-packard By any chance, have you had a chance to figure out what is wrong?

I don't have any easy way to test on Windows or Mac OS, alas. We'll need help with someone able to get a verbose log of the build process that includes the full gcc command line run with -v so we can see what happened.

[stephanosio]

It looks like --with-headers parameter is being specified twice while configuring libstdc++ for picolibc -- first time with the newlib header path and second time with the picolibc header path. This happens for all host OSes, including Linux.

From aarch64-zephyr-elf build log for x86-64 macOS:

[INFO ]  Installing libstdc++ picolibc
[DEBUG]    Entering '/Volumes/Workspace/build/.build/HOST-x86_64-apple-darwin/aarch64-zephyr-elf/build/build-cc-libstdcxx-picolibc'
[EXTRA]    Configuring libstdc++ for picolibc
[DEBUG]    Extra config passed: '--enable-stdio=stdio_pure --disable-wchar_t
--with-headers=/Volumes/Workspace/output/HOST-x86_64-apple-darwin/aarch64-zephyr-elf/aarch64-zephyr-elf/include
--with-headers=/Volumes/Workspace/output/HOST-x86_64-apple-darwin/aarch64-zephyr-elf/picolibc/include
--with-newlib
...

It seems the Linux host build ends up using the second/last --with-headers value, while macOS and Windows host builds somehow end up using the first one which is the newlib header path.

The same thing happens with the newlib-nano libstdc++ build; but, it did not cause any problems and went unnoticed because they are both newlibs and they both use _impure_ptr.

--with-headers is specified twice because CT_CC_SYSROOT_ARG is set to the default/newlib include path:

[DEBUG]    CT_CC_SYSROOT_ARG=--with-headers=/Volumes/Workspace/output/HOST-x86_64-apple-darwin/aarch64-zephyr-elf/aarch64-zephyr-elf/include

[stephanosio]

https://github.com/zephyrproject-rtos/crosstool-ng/pull/29 has managed to fix the multiple --with-headers issue and managed to fix the macos-aarch64 host build, which is built natively on an AArch64 macOS host.

Unfortunately, the problem seems to go a bit deeper than just that for Canadian cross builds such as macos-x86_64 and windows-x86_64, which are built from AArch64 macOS host and x86-64 Linux host, respectively.

Even with --with-headers=(PICOLIBC_INCLUDE_DIR), Canadian cross toolchain builds are not adding the picolibc include directory to the include paths when compiling libstdc++ source files.

"Include" parameters from ios_init.cc compile command line (full command line can be found here):

# macos-aarch64 (native)
...
-isystem /Volumes/Workspace/output/aarch64-zephyr-elf/picolibc/aarch64-zephyr-elf/include
-isystem /Volumes/Workspace/output/aarch64-zephyr-elf/picolibc/aarch64-zephyr-elf/sys-include
-I/Volumes/Workspace/build/.build/aarch64-zephyr-elf/src/gcc/libstdc++-v3/../libgcc
-I/Volumes/Workspace/build/.build/aarch64-zephyr-elf/build/build-cc-libstdcxx-picolibc/aarch64-zephyr-elf/libstdc++-v3/include/aarch64-zephyr-elf
-I/Volumes/Workspace/build/.build/aarch64-zephyr-elf/build/build-cc-libstdcxx-picolibc/aarch64-zephyr-elf/libstdc++-v3/include
-I/Volumes/Workspace/build/.build/aarch64-zephyr-elf/src/gcc/libstdc++-v3/libsupc++
...
-idirafter /Volumes/Workspace/output/aarch64-zephyr-elf/aarch64-zephyr-elf/include

# macos-x86_64 (Canadian)
...
-I/Volumes/Workspace/build/.build/HOST-x86_64-apple-darwin/aarch64-zephyr-elf/src/gcc/libstdc++-v3/../libgcc
-I/Volumes/Workspace/build/.build/HOST-x86_64-apple-darwin/aarch64-zephyr-elf/build/build-cc-libstdcxx-picolibc/aarch64-zephyr-elf/libstdc++-v3/include/aarch64-zephyr-elf
-I/Volumes/Workspace/build/.build/HOST-x86_64-apple-darwin/aarch64-zephyr-elf/build/build-cc-libstdcxx-picolibc/aarch64-zephyr-elf/libstdc++-v3/include
-I/Volumes/Workspace/build/.build/HOST-x86_64-apple-darwin/aarch64-zephyr-elf/src/gcc/libstdc++-v3/libsupc++
-I/Volumes/Workspace/output/HOST-x86_64-apple-darwin/aarch64-zephyr-elf/include
...
-idirafter /Volumes/Workspace/output/HOST-x86_64-apple-darwin/aarch64-zephyr-elf/aarch64-zephyr-elf/include

The same thing happens for windows-x86_64 host (Canadian cross) toolchain build.

Apart from the above, there is now another issue causing tests/lib/c_lib/thrd/libraries.libc.c11_threads.newlib build failure for all hosts, including Linux -- this was not failing prior to https://github.com/zephyrproject-rtos/crosstool-ng/pull/29:

[78/189] Building C object zephyr/lib/posix/options/CMakeFiles/lib__posix__options.dir/fs.c.obj
FAILED: zephyr/lib/posix/options/CMakeFiles/lib__posix__options.dir/fs.c.obj 
ccache /__w/sdk-ng/sdk-ng/tools/zephyr-sdk-0.16.8-19-g5402330/aarch64-zephyr-elf/bin/aarch64-zephyr-elf-gcc -DKERNEL -DK_HEAP_MEM_POOL_SIZE=1024 -DTC_RUNID=bfb4a20048f4d148752423470a596c16 -D_ANSI_SOURCE -D__LINUX_ERRNO_EXTENSIONS__ -D__ZEPHYR__=1 -I/__w/sdk-ng/sdk-ng/zephyrproject/zephyr/kernel/include -I/__w/sdk-ng/sdk-ng/zephyrproject/zephyr/arch/arm64/include -I/__w/sdk-ng/sdk-ng/test/twister-out/qemu_cortex_a53/tests/lib/c_lib/thrd/libraries.libc.c11_threads.newlib_nano/zephyr/include/generated/zephyr -I/__w/sdk-ng/sdk-ng/zephyrproject/zephyr/include -I/__w/sdk-ng/sdk-ng/test/twister-out/qemu_cortex_a53/tests/lib/c_lib/thrd/libraries.libc.c11_threads.newlib_nano/zephyr/include/generated -I/__w/sdk-ng/sdk-ng/zephyrproject/zephyr/soc/arm -I/__w/sdk-ng/sdk-ng/zephyrproject/zephyr/lib/libc/newlib/include -I/__w/sdk-ng/sdk-ng/zephyrproject/zephyr/include/zephyr/posix -I/__w/sdk-ng/sdk-ng/zephyrproject/zephyr/soc/arm/qemu_cortex_a53/. -I/__w/sdk-ng/sdk-ng/zephyrproject/zephyr/subsys/testsuite/include -I/__w/sdk-ng/sdk-ng/zephyrproject/zephyr/subsys/testsuite/coverage -I/__w/sdk-ng/sdk-ng/zephyrproject/zephyr/subsys/testsuite/ztest/include -isystem /__w/sdk-ng/sdk-ng/zephyrproject/zephyr/lib/libc/common/include -Os -DNDEBUG -Wshadow -fno-strict-aliasing -Werror -Os -imacros /__w/sdk-ng/sdk-ng/test/twister-out/qemu_cortex_a53/tests/lib/c_lib/thrd/libraries.libc.c11_threads.newlib_nano/zephyr/include/generated/zephyr/autoconf.h -fno-common -g -gdwarf-4 -fdiagnostics-color=always -mcpu=cortex-a53 -mabi=lp64 --sysroot=/__w/sdk-ng/sdk-ng/tools/zephyr-sdk-0.16.8-19-g5402330/aarch64-zephyr-elf/aarch64-zephyr-elf -imacros /__w/sdk-ng/sdk-ng/zephyrproject/zephyr/include/zephyr/toolchain/zephyr_stdint.h -Wall -Wformat -Wformat-security -Wno-format-zero-length -Wdouble-promotion -Wno-pointer-sign -Wpointer-arith -Wexpansion-to-defined -Wno-unused-but-set-variable -Werror=implicit-int -fno-pic -fno-pie -fno-asynchronous-unwind-tables -fno-reorder-functions --param=min-pagesize=0 -fno-defer-pop -fmacro-prefix-map=/__w/sdk-ng/sdk-ng/zephyrproject/zephyr/tests/lib/c_lib/thrd=CMAKE_SOURCE_DIR -fmacro-prefix-map=/__w/sdk-ng/sdk-ng/zephyrproject/zephyr=ZEPHYR_BASE -fmacro-prefix-map=/__w/sdk-ng/sdk-ng/zephyrproject=WEST_TOPDIR -ffunction-sections -fdata-sections -mno-outline-atomics -moverride=tune=no_ldp_stp_qregs -specs=nano.specs -D_POSIX_THREADS -std=c99 -MD -MT zephyr/lib/posix/options/CMakeFiles/lib__posix__options.dir/fs.c.obj -MF zephyr/lib/posix/options/CMakeFiles/lib__posix__options.dir/fs.c.obj.d -o zephyr/lib/posix/options/CMakeFiles/lib__posix__options.dir/fs.c.obj -c /__w/sdk-ng/sdk-ng/zephyrproject/zephyr/lib/posix/options/fs.c
In file included from /__w/sdk-ng/sdk-ng/zephyrproject/zephyr/lib/posix/options/fs.c:13:
/__w/sdk-ng/sdk-ng/zephyrproject/zephyr/include/zephyr/posix/dirent.h:23:21: error: 'PATH_MAX' undeclared here (not in a function)
   23 |         char d_name[PATH_MAX + 1];
      |                     ^~~~~~~~
In file included from /__w/sdk-ng/sdk-ng/zephyrproject/zephyr/include/zephyr/toolchain.h:50,
                 from /__w/sdk-ng/sdk-ng/zephyrproject/zephyr/include/zephyr/kernel_includes.h:23,
                 from /__w/sdk-ng/sdk-ng/zephyrproject/zephyr/include/zephyr/kernel.h:17,
                 from /__w/sdk-ng/sdk-ng/zephyrproject/zephyr/lib/posix/options/fs.c:10:
/__w/sdk-ng/sdk-ng/zephyrproject/zephyr/include/zephyr/toolchain/gcc.h:87:51: error: expression in static assertion is not an integer
   87 | #define BUILD_ASSERT(EXPR, MSG...) _Static_assert((EXPR), "" MSG)
      |                                                   ^
/__w/sdk-ng/sdk-ng/zephyrproject/zephyr/lib/posix/options/fs.c:22:1: note: in expansion of macro 'BUILD_ASSERT'
   22 | BUILD_ASSERT(PATH_MAX >= MAX_FILE_NAME, "PATH_MAX is less than MAX_FILE_NAME");
      | ^~~~~~~~~~~~

[stephanosio]

Unfortunately, the problem seems to go a bit deeper than just that for Canadian cross builds such as macos-x86_64 and windows-x86_64, which are built from AArch64 macOS host and x86-64 Linux host, respectively.

Even with --with-headers=(PICOLIBC_INCLUDE_DIR), Canadian cross toolchain builds are not adding the picolibc include directory to the include paths when compiling libstdc++ source files.

So, it turns out that GCC build system expects "we'll be searching the right directories for the previously-installed cross compiler" for Canadian cross compilers and does not bother adding the relevant target directories to the target flags, which is indeed the correct behaviour if we are building libstdc++ for the primary/default libc, which is currently newlib.

# On Canadian crosses, we'll be searching the right directories for
# the previously-installed cross compiler, so don't bother to add
# flags for directories within the install tree of the compiler
# being built; programs in there won't even run.
if test "${build}" = "${host}" && test -d ${srcdir}/gcc; then
  # Search for pre-installed headers if nothing else fits.
  FLAGS_FOR_TARGET=$FLAGS_FOR_TARGET' -B$(build_tooldir)/bin/ -B$(build_tooldir)/lib/ -isystem $(build_tooldir)/include -isystem $(build_tooldir)/sys-include'
fi

The problem is that, for the "companion libs" such as newlib-nano and picolibc, GCC will not pick them up unless the correct spec file is specified (or the include and library paths are manually specified as done above for non-Canadian builds).

The correct fix in this case would be, for each companion libc, to specify the --specs parameter for the companion libc in the CFLAGS when building the corresponding libstdc++.

[stephanosio]

Apart from the above, there is now another issue causing tests/lib/c_lib/thrd/libraries.libc.c11_threads.newlib build failure for all hosts, including Linux -- this was not failing prior to zephyrproject-rtos/crosstool-ng#29:

This likely has to do with the fact that --with-headers is no longer specified when building the final GCC compiler after https://github.com/zephyrproject-rtos/crosstool-ng/pull/29 -- the build process was (arguably incorrectly) relying on CT_CC_SYSROOT_ARG to provide --with-headers parameter for the final GCC compiler.

[EXTRA]    Configuring final gcc compiler
[DEBUG]    Extra config passed: '--with-newlib --enable-threads=no --disable-shared --with-pkgversion=Zephyr SDK 0.16.8-19-g5402330 --with-bugurl=https://github.com/zephyrproject-rtos/sdk-ng/issues --enable-__cxa_atexit --disable-libgomp --disable-libmudflap --disable-libmpx --disable-libssp --disable-libquadmath --disable-libquadmath-support --disable-libstdcxx-verbose --with-gmp=/Volumes/Workspace/build/.build/aarch64-zephyr-elf/buildtools --with-mpfr=/Volumes/Workspace/build/.build/aarch64-zephyr-elf/buildtools --with-mpc=/Volumes/Workspace/build/.build/aarch64-zephyr-elf/buildtools --with-isl=/Volumes/Workspace/build/.build/aarch64-zephyr-elf/buildtools --enable-lto --enable-target-optspace --disable-nls --enable-multiarch'

[stephanosio]

Apart from the above, there is now another issue causing tests/lib/c_lib/thrd/libraries.libc.c11_threads.newlib build failure for all hosts, including Linux -- this was not failing prior to zephyrproject-rtos/crosstool-ng#29:

This likely has to do with the fact that --with-headers is no longer specified when building the final GCC compiler after zephyrproject-rtos/crosstool-ng#29 -- the build process was (arguably incorrectly) relying on CT_CC_SYSROOT_ARG to provide --with-headers parameter for the final GCC compiler.

This is indeed correct and will be fixed by https://github.com/zephyrproject-rtos/crosstool-ng/pull/29/commits/53b5327d821a6b6949fa455f14a9631146af897c.

[stephanosio]

The problem is that, for the "companion libs" such as newlib-nano and picolibc, GCC will not pick them up unless the correct spec file is specified (or the include and library paths are manually specified as done above for non-Canadian builds).

The correct fix in this case would be, for each companion libc, to specify the --specs parameter for the companion libc in the CFLAGS when building the corresponding libstdc++.

GCC build system simply refuses to work with the specs file specified in the target flags.

After further investigation, the above excerpt selectively adding the target directories to the target flags for non-Canadian cross compilers should only apply to the -B parameters, which specify the GCC executable base path -- there is no reason why the target include path parameters such as -isystem should be excluded because they do not have anything to do with the host and build environments (note that the premise for excluding the target directory parameters for Canadian builds is that the build system cannot execute the host binaries).

This was worked around by fixing the GCC itself to ensure that the target include directories under exec_prefix are always added as -isystem, regardless of whether building a Canadian cross compiler.

For more details, see https://github.com/zephyrproject-rtos/gcc/pull/36.