Linking fails on Ubuntu 14.04 when using zlib

[WebDrake]

A simple test program using zlib:

void main ()
{
    import std.zlib : compress, uncompress;
    auto src = "The quick brown fox jumps over the lazy dog\n"
             ~ "The quick brown fox jumps over the lazy dog";

    ubyte[] dst;
    ubyte[] result;

    dst = compress(src);
    result = cast(ubyte[])uncompress(dst);
    assert(result == src);
    import std.stdio : writeln;
    writeln(cast(char[])result);
}

Compiling this on 14.04 fails at the link stage with the following message:

$ ldc2 -v zlibtest.d 
binary    /snap/ldc2/x1/bin/ldc2
version   1.1.1-beta1 (DMD v2.071.2, LLVM 3.9.1)
config    /snap/ldc2/x1/etc/ldc2.conf (x86_64-unknown-linux-gnu)
predefs   LDC all D_Version2 assert X86_64 D_InlineAsm_X86_64 D_HardFloat
          LittleEndian D_LP64 D_PIC linux Posix CRuntime_Glibc LDC_LLVM_309
parse     zlibtest
importall zlibtest
import    object    (/snap/ldc2/x1/bin/../include/d/ldc/object.d)
semantic  zlibtest
entry     main          zlibtest.d
semantic2 zlibtest
semantic3 zlibtest
import    std.zlib  (/snap/ldc2/x1/bin/../include/d/std/zlib.d)
import    etc.c.zlib    (/snap/ldc2/x1/bin/../include/d/etc/c/zlib.d)
import    core.stdc.config  (/snap/ldc2/x1/bin/../include/d/core/stdc/config.d)
import    std.stdio (/snap/ldc2/x1/bin/../include/d/std/stdio.d)
import    core.stdc.stdio   (/snap/ldc2/x1/bin/../include/d/core/stdc/stdio.d)
import    core.stdc.stdint  (/snap/ldc2/x1/bin/../include/d/core/stdc/stdint.d)
import    core.stdc.stddef  (/snap/ldc2/x1/bin/../include/d/core/stdc/stddef.d)
import    core.stdc.signal  (/snap/ldc2/x1/bin/../include/d/core/stdc/signal.d)
import    core.stdc.wchar_  (/snap/ldc2/x1/bin/../include/d/core/stdc/wchar_.d)
import    core.stdc.stdarg  (/snap/ldc2/x1/bin/../include/d/core/stdc/stdarg.d)
import    core.stdc.time    (/snap/ldc2/x1/bin/../include/d/core/stdc/time.d)
import    core.sys.posix.sys.types  (/snap/ldc2/x1/bin/../include/d/core/sys/posix/sys/types.d)
import    core.sys.posix.config (/snap/ldc2/x1/bin/../include/d/core/sys/posix/config.d)
import    std.typecons  (/snap/ldc2/x1/bin/../include/d/std/typecons.d)
import    std.meta  (/snap/ldc2/x1/bin/../include/d/std/meta.d)
import    std.traits    (/snap/ldc2/x1/bin/../include/d/std/traits.d)
import    std.typetuple (/snap/ldc2/x1/bin/../include/d/std/typetuple.d)
import    std.stdiobase (/snap/ldc2/x1/bin/../include/d/std/stdiobase.d)
import    std.range.primitives  (/snap/ldc2/x1/bin/../include/d/std/range/primitives.d)
import    std.random    (/snap/ldc2/x1/bin/../include/d/std/random.d)
import    std.conv  (/snap/ldc2/x1/bin/../include/d/std/conv.d)
import    std.ascii (/snap/ldc2/x1/bin/../include/d/std/ascii.d)
import    core.stdc.errno   (/snap/ldc2/x1/bin/../include/d/core/stdc/errno.d)
import    core.stdc.string  (/snap/ldc2/x1/bin/../include/d/core/stdc/string.d)
import    std.exception (/snap/ldc2/x1/bin/../include/d/std/exception.d)
import    std.range (/snap/ldc2/x1/bin/../include/d/std/range/package.d)
import    std.range.interfaces  (/snap/ldc2/x1/bin/../include/d/std/range/interfaces.d)
import    std.array (/snap/ldc2/x1/bin/../include/d/std/array.d)
import    std.functional    (/snap/ldc2/x1/bin/../include/d/std/functional.d)
import    std.algorithm (/snap/ldc2/x1/bin/../include/d/std/algorithm/package.d)
import    std.algorithm.comparison  (/snap/ldc2/x1/bin/../include/d/std/algorithm/comparison.d)
import    std.algorithm.iteration   (/snap/ldc2/x1/bin/../include/d/std/algorithm/iteration.d)
import    std.algorithm.mutation    (/snap/ldc2/x1/bin/../include/d/std/algorithm/mutation.d)
import    std.algorithm.setops  (/snap/ldc2/x1/bin/../include/d/std/algorithm/setops.d)
import    std.algorithm.sorting (/snap/ldc2/x1/bin/../include/d/std/algorithm/sorting.d)
import    std.algorithm.searching   (/snap/ldc2/x1/bin/../include/d/std/algorithm/searching.d)
import    std.utf   (/snap/ldc2/x1/bin/../include/d/std/utf.d)
import    core.internal.string  (/snap/ldc2/x1/bin/../include/d/core/internal/string.d)
import    core.bitop    (/snap/ldc2/x1/bin/../include/d/core/bitop.d)
import    ldc.intrinsics    (/snap/ldc2/x1/bin/../include/d/ldc/intrinsics.di)
code      zlibtest
/usr/bin/gcc zlibtest.o -o zlibtest -L/snap/ldc2/x1/bin/../lib -lphobos2-ldc -ldruntime-ldc -Wl,--gc-sections -lrt -ldl -lpthread -lm -m64 
/usr/bin/ld: /snap/ldc2/x1/bin/../lib/libphobos2-ldc.a(zutil.c.o): unrecognized relocation (0x2a) in section `.text'
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
Error: /usr/bin/gcc failed with status: 1

The same issue is not observed on 16.04 or 16.10.

[WebDrake]

@klickverbot assuming I built zlib from source as part of the snap, what's the most convenient way to let the LDC build process know about that library for building phobos?

[dnadlinger]

Phobos includes its own copy of zlib, so that shouldn't be it. Which platform is used for actually building the snaps? 16.04 by any chance? This might be another toolchain/ABI version issue.

[WebDrake]

Phobos includes its own copy of zlib, so that shouldn't be it.

I thought it needed a pre-existing zlib library to statically link when phobos is built? I was assuming that might be the trouble, since I'm just using the zlib1g-dev package on Ubuntu.

[dnadlinger]

I thought it needed a pre-existing zlib library to statically link when phobos is built?

There is a copy of zlib in etc/c/zlib. The error message is also from the static phobos library, not some external copy of zlib.

[WebDrake]

Ah, interesting. I was still going by the (probably very out of date) dependency indications here: https://wiki.dlang.org/Building_LDC_from_source#Prerequisites

I'll edit the snap package to remove the zlib1g-dev build dependency and see if any change results -- probably not, but can't hurt as a first step.

[dnadlinger]

I was still going by the (probably very out of date) dependency indications here

That's for LDC itself – LLVM (sometimes (?)) depends on zlib as well.

[WebDrake]

Ah, OK, thanks. Then I'll check things out again on the snapcraft side and see what comes up.

[dnadlinger]

A quick online search turns up a few results that suggest that it is indeed a new relocation type added fairly recently, and binutils might not support it. Did you try the binary packages (which are actually built on 14.04)?

[WebDrake]

No; I'll give that a go and report back on how they behave.

[WebDrake]

Sorry for delayed response. The current LDC 1.1.1 download from GitHub works fine, which would support the hypothesis that it's a difference in binutils that is responsible: 14.04 uses binutils 2.24 whereas 16.04 uses 2.26.

[WebDrake]

@klickverbot the snapcraft folks have suggested passing the flags -Wa,-mrelax-relocations=no to GCC when it builds zlib. Can you recommend the best way of achieving this? Is it possible just by adding some options when invoking cmake or would it require some tweaks to the build process?

[dnadlinger]

@WebDrake: We use the regular CMake functions for building C sources, so you can just do cmake -DCMAKE_C_FLAGS=… (or whatever else is appropriate).

[WebDrake]

OK, cool. Are there any predefined ones that I should take care not to overwrite? Or will the existing scripts just add anything necessary to whatever I define?

[dnadlinger]

All the flag handling should be additive. The only thing to watch out for would be distro-specific defaults (which I think you should be able to verify using ccmake . on a default build (t for advanced flags)).

[WebDrake]

There don't seem to be any C_FLAGS defined, according to ccmake. The weird thing is nothing shows up even if I set things up with cmake -DCMAKE_C_FLAGS="-Wa,-mrelax-relocations=no".

Still, might as well try things out with this and see where it goes ;-)

[dnadlinger]

The weird thing is nothing shows up

Did you re-run CMake in a fresh directory?

[WebDrake]

Did you re-run CMake in a fresh directory?

Yup.

[WebDrake]

To follow up on here: I've run a snapcraft cleanbuild with and without the -DCMAKE_C_FLAGS option, using a verbose build. Here's the result without the flags defined:

[ 50%] Building C object runtime/CMakeFiles/druntime-ldc-debug.dir/druntime/src/core/stdc/errno.c.o
cd /root/parts/ldc/build/runtime && /usr/bin/cc  -DLDC_WITH_PGO -I/root/parts/ldc/src/. -I/root/parts/ldc/src/ddmd -I/root/parts/ldc/src/ddmd/root -I/root/parts/ldc/build/ddmd -I/root/parts/ldc/src -isystem /root/parts/llvm/install/include  -O3 -DNDEBUG -fPIC   -o CMakeFiles/druntime-ldc-debug.dir/druntime/src/core/stdc/errno.c.o   -c /root/parts/ldc/src/runtime/druntime/src/core/stdc/errno.c

and with:

[ 56%] Building C object runtime/CMakeFiles/druntime-ldc-debug.dir/druntime/src/core/stdc/errno.c.o
cd /root/parts/ldc/build/runtime && /usr/bin/cc  -DLDC_WITH_PGO -I/root/parts/ldc/src/. -I/root/parts/ldc/src/ddmd -I/root/parts/ldc/src/ddmd/root -I/root/parts/ldc/build/ddmd -I/root/parts/ldc/src -I/root/parts/llvm/install/include  "-Wa,-mrelax-relocations=no"   -o CMakeFiles/druntime-ldc-debug.dir/druntime/src/core/stdc/errno.c.o   -c /root/parts/ldc/src/runtime/druntime/src/core/stdc/errno.c

... i.e. for the 'regular' build the C flags are -O3 -DNDEBUG -fPIC and for the build with this patch, it's "-Wa,-mrelax-relocations=no". This would explain why I'm getting the PIC-related errors on 16.10.

It should be straightforward enough to add the extra flags, but I'm wondering if there is a way to avoid this (given the earlier discussion about how C flags should be additive).

[WebDrake]

Doing a non-clean build (i.e. using the snapcraft command alone) results in:

[ 49%] Building C object runtime/CMakeFiles/druntime-ldc-debug.dir/druntime/src/core/stdc/errno.c.o
cd /home/joseph/code/snap/ldc2.snap/parts/ldc/build/runtime && /usr/bin/cc  -DLDC_WITH_PGO -I/home/joseph/code/snap/ldc2.snap/parts/ldc/src/. -I/home/joseph/code/snap/ldc2.snap/parts/ldc/src/ddmd -I/home/joseph/code/snap/ldc2.snap/parts/ldc/src/ddmd/root -I/home/joseph/code/snap/ldc2.snap/parts/ldc/build/ddmd -I/home/joseph/code/snap/ldc2.snap/parts/ldc/src -I/home/joseph/code/snap/ldc2.snap/parts/llvm/install/include  "-Wa,-mrelax-relocations=no"   -o CMakeFiles/druntime-ldc-debug.dir/druntime/src/core/stdc/errno.c.o   -c /home/joseph/code/snap/ldc2.snap/parts/ldc/src/runtime/druntime/src/core/stdc/errno.c

... i.e. also missing the optimization flags. So I suspect the best way forward with this is to just put all the wanted C flags directly into the snap package definition.

[dnadlinger]

Maybe it's just me misunderstanding how CMAKE_C_FLAGS works and somehow not noticing in a couple of years. It would be good to track down where exactly the flags are overwritten (e.g. by having a look at the CMake cache, directly or using ccmake).