Ubuntu 16.10 linker failures due to PIE by default (relocation R_X86_64_32S … can not be used)

[ximion]

Hi! I am trying to make appstream-generator compile with LDC, and also make it your new default D compiler on Debian. Unfortunately, compiling fails with

collect2: fatal error: ld terminated with signal 6 [Abgebrochen], core dumped
compilation terminated.
/usr/bin/ld.bfd.real: ld.bfd.real: wcsrtombs.c:94: __wcsrtombs: Zusicherung »data.__outbuf[-1] == '\0'« nicht erfüllt.
Error: /usr/bin/gcc failed with status: 1

(assert failure)

When running with LANG=C, I get a different error:

/usr/bin/ld.bfd.real: .dub/obj/app.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
.dub/obj/app.o: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
Error: /usr/bin/gcc failed with status: 1
/tmp/ldc2-1.0.0-linux-x86_64/bin/ldc2 failed with exit code 1.

A similar error also appears when trying to compile Terminix

terminix ~master: building configuration "dynamic"...
/usr/bin/ld.bfd.real: .dub/obj/app.o: Die Umlagerung von 
.dub/obj/app.o: error adding symbols: Ungültiger Wert
collect2: error: ld returned 1 exit status
Error: /usr/bin/gcc failed with status: 1
/tmp/ldc2-1.0.0-linux-x86_64/bin/ldc2 failed with exit code 1.

terminix ~master: building configuration "dynamic"...
/usr/bin/ld.bfd.real: .dub/obj/app.o: relocation R_X86_64_32 against `_D3app4mainFAAyaZ12__dgliteral2MFNaNbNfZAya' can not be used when making a shared object; recompile with -fPIC
.dub/obj/app.o: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
Error: /usr/bin/gcc failed with status: 1

I am running on an up-to-date Debian, compiling is done with dub. GCC version is 5.4, the issue appears when using the precompiled sources as well as when using manually compiled LDC binaries. In both cases, shared libraries are used, while asgen uses system shared libs written in C, Terminix uses a D shared library (gtkd-3.so).

Since issues of this kind appear even with a minimal hello-world application, I expect this issue to be some toolchain mismatch, or maybe even a bug in dub. In that case, I will reassign the issue.

[kinke]

Seems related: http://stackoverflow.com/questions/35380213/gcc-link-error-wcsrtombs-zusicherung-data-outbuf-1-0-nicht-erf%C3%BCl

[dnadlinger]

The German error message might well be an ld.bfd diagnostics bug (?), so let's leave that aside for now.

The actual error seems to be […] can not be used when making a shared object; recompile with -fPIC – are you trying to mix static and dynamic libraries or something along these lines? What is the actual compiler/linker command used? (dub -v, then ldc -v)

[ximion]

Using the precompiled binaries from Github. dub -v and ldc2 -v:

/tmp/ldc2-1.0.0-linux-x86_64/bin/ldc2 -of.dub/build/application-debug-linux.posix-x86_64-ldc_0-86916BD123F7D97668C6BDFDCD102884/test -d-debug -g -v -w -oq -od=.dub/obj -d-version=Have_test -Isource/ source/app.d
binary    /tmp/ldc2-1.0.0-linux-x86_64/bin/ldc2
version   1.0.0 (DMD v2.070.2, LLVM 3.8.0)
config    /tmp/ldc2-1.0.0-linux-x86_64/etc/ldc2.conf
predefs   Have_test LDC all D_Version2 assert X86_64 D_InlineAsm_X86_64
          D_HardFloat LittleEndian D_LP64 linux Posix CRuntime_Glibc
          LDC_LLVM_308
parse     app
importall app
import    object        (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/ldc/object.d)
import    std.stdio     (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/stdio.d)
import    core.stdc.stdio       (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/core/stdc/stdio.d)
import    core.stdc.config      (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/core/stdc/config.d)
import    core.stdc.stdarg      (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/core/stdc/stdarg.d)
import    core.stdc.stdint      (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/core/stdc/stdint.d)
import    core.stdc.stddef      (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/core/stdc/stddef.d)
import    core.stdc.signal      (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/core/stdc/signal.d)
import    core.stdc.wchar_      (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/core/stdc/wchar_.d)
import    core.stdc.time        (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/core/stdc/time.d)
import    core.sys.posix.sys.types      (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/core/sys/posix/sys/types.d)
import    core.sys.posix.config (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/core/sys/posix/config.d)
import    std.typecons  (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/typecons.d)
import    std.meta      (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/meta.d)
import    std.traits    (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/traits.d)
import    std.typetuple (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/typetuple.d)
import    std.stdiobase (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/stdiobase.d)
import    std.range.primitives  (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/range/primitives.d)
semantic  app
import    core.stdc.errno       (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/core/stdc/errno.d)
import    core.stdc.string      (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/core/stdc/string.d)
entry     main          source/app.d
semantic2 app
semantic3 app
import    std.exception (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/exception.d)
import    std.range     (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/range/package.d)
import    std.range.interfaces  (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/range/interfaces.d)
import    std.array     (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/array.d)
import    std.functional        (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/functional.d)
import    std.algorithm (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/algorithm/package.d)
import    std.algorithm.comparison      (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/algorithm/comparison.d)
import    std.algorithm.iteration       (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/algorithm/iteration.d)
import    std.algorithm.mutation        (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/algorithm/mutation.d)
import    std.algorithm.setops  (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/algorithm/setops.d)
import    std.algorithm.sorting (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/algorithm/sorting.d)
import    std.algorithm.searching       (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/algorithm/searching.d)
import    std.utf       (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/std/utf.d)
import    core.internal.string  (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/core/internal/string.d)
import    core.bitop    (/tmp/ldc2-1.0.0-linux-x86_64/bin/../import/core/bitop.d)
code      app
/usr/bin/gcc .dub/obj/.dub/build/application-debug-linux.posix-x86_64-ldc_0-86916BD123F7D97668C6BDFDCD102884/test.o -o .dub/build/application-debug-linux.posix-x86_64-ldc_0-86916BD123F7D97668C6BDFDCD102884/test -L/tmp/ldc2-1.0.0-linux-x86_64/bin/../lib -L/tmp/ldc2-1.0.0-linux-x86_64/bin/../lib32 -Xlinker --no-warn-search-mismatch -lphobos2-ldc -ldruntime-ldc -Wl,--gc-sections -lrt -ldl -lpthread -lm -m64 
/usr/bin/ld.bfd.real: .dub/obj/.dub/build/application-debug-linux.posix-x86_64-ldc_0-86916BD123F7D97668C6BDFDCD102884/test.o: relocation R_X86_64_32 against `.rodata.str1.16' can not be used when making a shared object; recompile with -fPIC
.dub/obj/.dub/build/application-debug-linux.posix-x86_64-ldc_0-86916BD123F7D97668C6BDFDCD102884/test.o: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
Error: /usr/bin/gcc failed with status: 1
FAIL .dub/build/application-debug-linux.posix-x86_64-ldc_0-86916BD123F7D97668C6BDFDCD102884/ test executable
/tmp/ldc2-1.0.0-linux-x86_64/bin/ldc2 failed with exit code 1.

[dnadlinger]

Okay, so first off, we can be pretty sure this is not a Dub issue, unless it has started setting some environment variables read by GCC since I last looked. There is only a single D file involved, and no user D libraries where something could go wrong in the way Dub sets up the linking steps.

There does not seem to be an obvious reason for the actual error message, though, unless Debian's GCC is doing something funky to the default linker flags. Does it work if you switch your system to use ld.gold instead? (You can either add a ld symlink to your PATH or get the actual linker command from gcc and change it.) Does it still occur when removing -Wl,--gc-sections? With -disable-linker-strip-dead? When building LDC from source?

[ximion]

Maybe an interesting data point: We got Terminix to compile in https://github.com/gnunn1/terminix/issues/387 simply by exchanging ldc with ldmd...

I will try the other suggestions tomorrow :)

[ximion]

FTR, running on Ubuntu Yakkety now, which shows the same behavior. The issue seems to be gone now in Debian unstable, with the latest, recompiled LDC.

Okay, with gold I get (almost) the same result:

ldc2 source/app.d
/usr/bin/ld.gold.real: error: app.o: requires dynamic R_X86_64_32 reloc which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold.real: error: app.o: requires dynamic R_X86_64_PC32 reloc against '_D3std5stdio13trustedStdoutFNdNeZS3std5stdio4File' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold.real: error: app.o: requires dynamic R_X86_64_32 reloc which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold.real: error: app.o: requires dynamic R_X86_64_32 reloc which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold.real: error: app.o: requires dynamic R_X86_64_PC32 reloc against 'fwrite' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold.real: error: app.o: requires dynamic R_X86_64_PC32 reloc against '_d_newclass' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold.real: error: app.o: requires dynamic R_X86_64_32 reloc which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold.real: error: app.o: requires dynamic R_X86_64_PC32 reloc against 'fputc_unlocked' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold.real: error: app.o: requires dynamic R_X86_64_PC32 reloc against 'fputwc_unlocked' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold.real: error: app.o: requires dynamic R_X86_64_PC32 reloc against 'fputc_unlocked' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold.real: error: app.o: requires dynamic R_X86_64_PC32 reloc against 'fputwc_unlocked' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold.real: error: app.o: requires unsupported dynamic reloc 11; recompile with -fPIC
/usr/bin/ld.gold.real: error: app.o: requires unsupported dynamic reloc 11; recompile with -fPIC
/usr/bin/ld.gold.real: error: app.o: requires dynamic R_X86_64_PC32 reloc against '_d_run_main' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold.real: error: app.o: requires dynamic R_X86_64_32 reloc against '_d_eh_personality' which may overflow at runtime; recompile with -fPIC
collect2: error: ld returned 1 exit status
Error: /usr/bin/gcc failed with status: 1

Same result with -disable-linker-strip-dead The GCC config is:

Using built-in specs.
COLLECT_GCC=/usr/bin/gcc-5.real
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1)

Recompiling LDC on Ubuntu too now, let's see if that fixes it.

[ximion]

LDC compiling fails at

cd /tmp/ldc/ldc-1.1.0 && /usr/bin/ldmd2 -wi -O -inline -release -J/tmp/ldc/ldc-1.1.0/ddmd -I/tmp/ldc/ldc-1.1.0/ddmd -of/tmp/ldc/ldc-1.1.0/build-static/ddmd/idgen ddmd/idgen.d

But if I add -fPIC to the falgs in the build_idgen macro, it seems to compile properly.

[ximion]

After applying this (hackish) patch to LDC, I could successfully compile it:

--- ldc-1.1.0.orig/CMakeLists.txt
+++ ldc-1.1.0/CMakeLists.txt
@@ -316,6 +316,7 @@ endmacro()
 # Build (generate executable) of D source
 macro(build_idgen  input_d  output_exec  extra_d_flags  link_flags  extra_deps)
     separate_arguments(FLAG_LIST WINDOWS_COMMAND "${D_COMPILER_FLAGS} ${extra_d_flags} ${link_flags}")
+    set(FLAG_LIST ${FLAG_LIST} -fPIC)
     add_custom_command(
         OUTPUT ${output_exec}
         COMMAND ${D_COMPILER} ${FLAG_LIST} -I${PROJECT_SOURCE_DIR}/${DDMDFE_PATH} -of${output_exec} ${input_d}
@@ -607,6 +608,7 @@ endif()

 # CONFIG generator expressions need to be repeated due to https://cmake.org/Bug/view.php?id=14353
 separate_arguments(LDC_FLAG_LIST WINDOWS_COMMAND "${tempVar} ${D_COMPILER_FLAGS} ${DDMD_DFLAGS} ${DDMD_LFLAGS}")
+set(LDC_FLAG_LIST ${LDC_FLAG_LIST} -fPIC)
 if(MSVC_IDE)
     add_custom_command(
         OUTPUT ${LDC_EXE_FULL}

Unfortunately, the resulting LDC is still unable to compile any application.

[markos]

probably related, but trying to bootstrap ldc on powerpc (32-bit), I'm getting this:

/root/ldc-1.1.0/build-static/lib/libldc.a(logger.cpp.o):(.debug_addr+0x60): undefined reference to `vwarning(Loc, char const*, __va_list_tag*)'
/usr/bin/ld: /usr/local/lib/libdruntime-ldc.a(sections_elf_shared.o): In function `_d_dso_registry':
/root/ldc.upstream/runtime/druntime/src/rt/sections_elf_shared.d:(.text._d_dso_registry[_d_dso_registry]+0x388): unresolvable R_PPC_REL24 relocation against symbol `__tls_get_addr_opt@@GLIBC_2.22'
/root/ldc.upstream/runtime/druntime/src/rt/sections_elf_shared.d:(.text._d_dso_registry[_d_dso_registry]+0x388): relocation truncated to fit: R_PPC_REL24 against symbol `__tls_get_addr_opt@@GLIBC_2.22' defined in .text section in /lib/powerpc-linux-gnu/ld.so.1
/usr/bin/ld: final link failed: Nonrepresentable section on output

any idea how to get past this? tried to append -relocation-model=pic to runtime/CMakeLists.txt to my bootstrap ldc (ltsmaster branch) but it didn't have any effect.

[ximion]

@klickverbot LDC running on Ubuntu Yakkety always fails, it's fine on Debian Testing - so there likely is some LLVM/toolchain mismatch on Yakkety, which doesn't get fixed by recompiling LDC on Yakkety. Really weird, I don't know LLVM enough to resolve this, unfortunately...

[dnadlinger]

@markos: How did you edit the CMake config file? If you just added the flag to the default settings, you CMake cache (i.e. build directory) might still be using the old flags. (I also took the liberty to add a code block to your comment for readability of the error message.)

[dnadlinger]

Looking at https://wiki.ubuntu.com/Security/Features (check the matrix for "Built as PIE"), they might have patched GCC to build everything as position-independent executables by default.

[dnadlinger]

https://wiki.ubuntu.com/SecurityTeam/PIE

[dnadlinger]

We can't possibly be the only compiler/language hitting this. We can of course just default to emitting position-independent objects on Ubuntu 16.10 and above, but for that we need to detect that circumstance somehow (I'd rather not implement distro version detection in LDC if avoidable). Another option would be to pass -no-pie to the linker when not building with -relocation-model=pic, but that option is not supported on older GCCs.

[dnadlinger]

Another option would of course be to just default to PIC code, like we e.g. do for OS X where it has also been required for a while. Some people might mind the slight performance hit, though.

[dnadlinger]

To end this monologue: I likely won't be able to follow up on this myself in the next couple of weeks, but now that we know what is going on, it should be straightforward to work out a solution. (@redstar: This will become quite critical once Ubuntu 16.10 hits the wild.)

[ximion]

@klickverbot Defaulting to PIE and PIC for all platforms which support it would be very great (for security and since it will likely also be default in e.g. newer Debian releases). Switching it of explicitly seems to be a good idea, IMHO.

[ximion]

Btw, I can confirm that when compiling with -relocation-model=pic all issues are gone.

[ximion]

This is fixed by the PR above, so everything works fine now :) Thanks to everyone who helped investigating (and ultimately resolving) this issue!

[es20490446e]

In the command line: cmake -DCMAKE_EXE_LINKER_FLAGS="-no-pie"

Or in CMakeList.txt: set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -no-pie")