[Freebsd] llvmlite libgcc_s.so.1 issue

[ddkn]

Hello everyone,

After trying to isolate a libgcc_s.so.1 loading issue for numba, it turns out that llvmlite was the culprit. The original issue can be seen here: https://github.com/numba/numba/issues/2001 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211347 https://forums.freebsd.org/threads/57080/

In short the issue seems that -rpath is not being set for libllvmlite.so, the /lib/libgcc_s.so.1 is from legacy gcc 4.2.1, which is used for backwards compatibility. Newer versions of gcc are found in /usr/local/lib/gcc{GCC_VER}/libgcc_s.so.1. Below is the ldd output of libllvmlite.so, pointing to the incorrect libgcc_s.so.1

ldd /usr/local/lib/python2.7/site-packages/llvmlite/binding/libllvmlite.so /usr/local/lib/python2.7/site-packages/llvmlite/binding/libllvmlite.so: librt.so.1 => /usr/lib/librt.so.1 (0x281f6000) libncurses.so.8 => /lib/libncurses.so.8 (0x281fc000) libthr.so.3 => /lib/libthr.so.3 (0x28239000) libz.so.6 => /lib/libz.so.6 (0x2825a000) libm.so.5 => /lib/libm.so.5 (0x2826e000) libc++.so.1 => /usr/lib/libc++.so.1 (0x28295000) libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x28343000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2835d000) libc.so.7 => /lib/libc.so.7 (0x2806f000)

Is there a way for rpath to be easily added to the setup.py script?

Thanks

PS. I am not using conda, I am using the FreeBSD ports system. This is because the conda installer hasn't been tested on FreeBSD or ported over. Aside from this rpath issue everything works just fine, loading numpy ahead of time curbs the issue, but this isn't correct to include numpy before hand.

[ddkn]

Okay, so I got everything working in the sense that as a user one can run:

$ ipython [1]: import numba [2]:

It was llvmlite, what I did was patch llvmlite-0.13.0/ffi/Makefile.freebsd with +LDFLAGS = $(LLVM_LDFLAGS) -shared -Wl,-rpath=/usr/local/lib/gcc48

However, hard coding in the specific gcc version isn't ideal. Is it possible to pass that information from an external variable in the build.py?

The reason why is, configurations like this with version numbers etc... are handled from the freebsd port Makefile

Also the ports system let me apply individual file patches, so a quick fix can work now. The above patch was done as such

$ cat /home/dvk/project/patch-ffi-Makefile.freebsd
--- ffi/Makefile.freebsd.orig 2016-09-03 17:26:16.399458000 +0000 +++ ffi/Makefile.freebsd 2016-09-03 17:26:38.057115000 +0000 @@ -1,7 +1,7 @@

CXX = clang++ -std=c++11 -stdlib=libc++ CXXFLAGS = $(LLVM_CXXFLAGS) -LDFLAGS = $(LLVM_LDFLAGS) +LDFLAGS = $(LLVM_LDFLAGS) -shared -Wl,-rpath=/usr/local/lib/gcc48 LIBS = $(LLVM_LIBS) INCLUDE = core.h SRC = assembly.cpp bitcode.cpp core.cpp initfini.cpp module.cpp value.cpp \

[ddkn]

What I have done so far to alleviate the issue is:

$ cat /home/dvk/project/patch-ffi-Makefile.freebsd --- ffi/Makefile.freebsd.orig 2016-09-03 17:26:16.399458000 +0000 +++ ffi/Makefile.freebsd 2016-09-03 17:26:38.057115000 +0000 @@ -1,7 +1,7 @@

CXX = clang++ -std=c++11 -stdlib=libc++ CXXFLAGS = $(LLVM_CXXFLAGS) -LDFLAGS = $(LLVM_LDFLAGS) +LDFLAGS = $(LLVM_LDFLAGS) -shared -Wl,-rpath=$(LOCALBASE)/lib/gcc%%_GCC_VER%% LIBS = $(LLVM_LIBS) INCLUDE = core.h SRC = assembly.cpp bitcode.cpp core.cpp initfini.cpp module.cpp value.cpp \

where in the port Makefile I put:

post-patch: ${REINPLACE_CMD} -e "s|%%_GCC_VER%%|${_GCC_VER}| "${WRKSRC}/ffi/Makefile.freebsd

This works for now, as it is a FreeBSD issue, but variables from the port Makefile weren't passed so the above post-patch had to be implemented.

It would be nice if there was a nicer way to pass -shared -Wl,-rpath=... from a flag/variable.

The commit can be seen here: https://github.com/davidkallie/ports/tree/master/devel/py-llvmlite https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211346

PS. The reason for choosing ${_GCC_VER} is due to the fortran option for port Makefiles has that variable predefined.

[pitrou]

It would be nice to know why Numba got the rpath right but not llvmlite. Does FreeBSD set that in a specific environment variable that llvmlite's Makefile would not be taking up?

[ddkn]

Well there are different environment variables that get set. Numba, which is python specific only handles them nicely, however in llvmlite, there is some hardcoded Makefiles. Specifically Makefile.freebsd, where the LDFLAGS is set using LDFLAGS=$(LLVM_LDFLAGS).

However, when I define some variables for buliding, the Makefile.freebsd does not pick them up -- which is strange. Also LLVM_LDFLAGS is hard set in ffi/build.py.

I will have to do some investigating still as to why that is, my fix isn't one that I like. It would be nice to either piggy back on LLVM_LDFLAGS, or something else.

As far as I can tell, llvmlite does not seem to respect passing -Wl,-rpath="" as numba does. What is odd is that ${LOCALBASE} is accepted by the Makefile.freebsd, but no other variable. ${LOCALBASE} is a definition for FreeBSD for /usr/local

[ddkn]

On another note, as of today llvmlite 0.13.0 and numba 0.28.1 have been accepted into FreeBSDs ports tree, for FreeBSD 10+, FreeBSD 11, and FreeBSD-CURRENT. Now users that have a recent version of ports can install numba as such:

$ cd /usr/ports/devel/py-numba $ sudo make $ sudo make install

The commits to the ports tree can be seen here: llvmlite-0.13.0: https://svnweb.freebsd.org/changeset/ports/421377 numba-0.28.1: https://svnweb.freebsd.org/changeset/ports/421379

[pitrou]

So, is there something llvmlite can do for this issue? I don't use FreeBSD so I'm unable to propose any change.

[ddkn]

Thats fair, in the mean time I'll track master and see if I can provide a patch.