search

gmarcais / Jellyfish

A fast multi-threaded k-mer counter
Other
463 stars 136 forks source link

SWIG wrappers fail to link on CentOS 6 #45

Closed xzy3 closed 8 years ago

xzy3 commented 8 years ago

I can't seem to get the Python swig wrappers to link on CentOS 6.

SWIG 3.0.7, gcc 4.4.7, Python 2.7.6.

I've googled for a while but nothing is turning up about how to fix this issue.

make[1]: Entering directory `Jellyfish-2.2.3/swig'
  CXX    python/python__jellyfish_la-swig_wrap.lo
  CXXLD  python/_jellyfish.la
/usr/bin/ld: lib/libpython2.7.a(abstract.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC
lib/libpython2.7.a: could not read symbols: Bad value
gmarcais commented 8 years ago

I usually see this error when --disable-shared was used during configuration of either the library or the compiler. In particular, 

gcc -v 2>&1 | grep disable-shared
python -c "import distutils.sysconfig; print(distutils.sysconfig.get_config_var('CONFIG_ARGS'))" | grep disable-shared

should return nothing.

If for some reason you cannot create a shared library, then you cannot create any binding. I think the binding has to be a shared library, which is loaded via dlopen by the interpreter.

xzy3 commented 8 years ago

They do return nothing.

gmarcais commented 8 years ago

Who compiled python? Is it the version installed by CentOS6? Or did you compile it yourself? The error is that your library is not compiled properly as a shared library (with -shared and -fPIC). From what I can see the error is with your Python installation.

xzy3 commented 8 years ago

It is a non-standard install, python was compiled with only the --prefix option everything else was kept as defaults. I don't have the rights to alter the install however. We have several other extension packages linked against it though (numpy, scipy, etc...). I just don't think anyone has ever used SWIG to generate code it's mostly C-Python/Cython that has been linked against it in the past. I guess Cython does a better job of this particular use case then SWIG.

gmarcais commented 8 years ago

I guess Cython does a better job of this particular use case then SWIG.

In truth, SWIG has nothing to do with it. SWIG simply creates the C/C++ code to wrap the functionality. It is not involved in the compilation and linking.

Can you post the output of make V=1?

xzy3 commented 8 years ago

Yes I understand that SWIG is more like (F)LEX/YACC/Bison than Cython really. But Cython has the build process built in, and so far this kind of interface code has 'just worked'. I'm not sure what magic it's pulling to make that happen though.

Making all in swig
make[2]: Entering directory `Jellyfish-2.2.3/swig'
~/.local/bin/swig -c++ -python -I../include -o python/swig_wrap.cpp jellyfish.i
make  all-am
make[3]: Entering directory `~/Jellyfish-2.2.3/swig'
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I..  -I/apps/x86_64/python/2.7.6/include/python2.7 -I../include   -std=c++0x  -MT python/python__jellyfish_la-swig_wrap.lo -MD -MP -MF python/.deps/python__jellyfish_la-swig_wrap.Tpo -c -o python/python__jellyfish_la-swig_wrap.lo `test -f 'python/swig_wrap.cpp' || echo './'`python/swig_wrap.cpp
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I/apps/x86_64/python/2.7.6/include/python2.7 -I../include -std=c++0x -MT python/python__jellyfish_la-swig_wrap.lo -MD -MP -MF python/.deps/python__jellyfish_la-swig_wrap.Tpo -c python/swig_wrap.cpp  -fPIC -DPIC -o python/.libs/python__jellyfish_la-swig_wrap.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I/apps/x86_64/python/2.7.6/include/python2.7 -I../include -std=c++0x -MT python/python__jellyfish_la-swig_wrap.lo -MD -MP -MF python/.deps/python__jellyfish_la-swig_wrap.Tpo -c python/swig_wrap.cpp -o python/python__jellyfish_la-swig_wrap.o >/dev/null 2>&1
mv -f python/.deps/python__jellyfish_la-swig_wrap.Tpo python/.deps/python__jellyfish_la-swig_wrap.Plo
/bin/sh ../libtool  --tag=CXX   --mode=link g++  -std=c++0x  -module -L/apps/x86_64/python/2.7.6/lib -lpython2.7  -o python/_jellyfish.la -rpath /apps/x86_64/python/2.7.6/lib/python2.7/site-packages/jellyfish python/python__jellyfish_la-swig_wrap.lo  ../libjellyfish-2.0.la 
libtool: link: g++ -shared -nostdlib /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/crti.o /usr/lib/gcc/x86_64-redhat-linux/4.4.7/crtbeginS.o  python/.libs/python__jellyfish_la-swig_wrap.o   -Wl,-rpath -Wl,~/Jellyfish-2.2.3/.libs -Wl,-rpath -Wl,~/.local/lib -L/apps/x86_64/python/2.7.6/lib -lpython2.7 ../.libs/libjellyfish-2.0.so -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-redhat-linux/4.4.7/crtendS.o /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/crtn.o    -Wl,-soname -Wl,_jellyfish.so.0 -o python/.libs/_jellyfish.so.0.0.0
/usr/bin/ld: /apps/x86_64/python/2.7.6/lib/libpython2.7.a(abstract.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC
/apps/x86_64/python/2.7.6/lib/libpython2.7.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make[3]: *** [python/_jellyfish.la] Error 1
make[3]: Leaving directory `~/Jellyfish-2.2.3/swig'
make[2]: *** [all] Error 2
make[2]: Leaving directory `~/Jellyfish-2.2.3/swig'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `~/Jellyfish-2.2.3'
make: *** [all] Error 2
gmarcais commented 8 years ago

Does /apps/x86_64/python/2.7.6/lib contain libpython2.7.so? Or some shared library (.so file)? If not, then I don't think I can use that version of python. If it does, then the Makefile/linker failed to find it and I have a bug.

xzy3 commented 8 years ago

No we only have the static libpython2.7.a there are plenty of other *.so extension-packages down in the library folder but not libpython. Well I was hoping this was going to be something easy I guess not.

gmarcais commented 8 years ago

Actually, I am wrong. My bad. You don't need libpython*.so to create a loadable module. By not including -Lpython2.7, automake/libtool happily creates a shared library that loads in Python just fine.

If you pull the latest develop branch (not the master branch), you should be able to compile and install the python binding. Do let me know if it works, in which case I make a new release.

Thank you for reporting that bug and helping me fix that bug. I learned something new. Good day!

xzy3 commented 8 years ago

Same result:

I also noticed this: /bin/sh ../libtool --tag=CXX --mode=link g++ -std=c++0x -module -L/apps/x86_64/python/2.7.6/lib -lpython2.7 -o python/_jellyfish.la -rpath /apps/x86_64/python/2.7.6/lib/python2.7/site-packages/jellyfish python/python__jellyfish_la-swig_wrap.lo ../libjellyfish-2.0.la

That -rpath won't actually work I'm installing to the prefix ~/.local ./configure --enable-swig --enable-python-binding --prefix=$(readlink -f ~/.local)

Result

make V=1
~/.local/bin/swig -c++ -python -I../include -o python/swig_wrap.cpp jellyfish.i
make  all-am
make[1]: Entering directory `~/builds/Jellyfish/swig'
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I..  -I/apps/x86_64/python/2.7.6/include/python2.7 -I../include   -std=c++0x  -MT python/python__jellyfish_la-swig_wrap.lo -MD -MP -MF python/.deps/python__jellyfish_la-swig_wrap.Tpo -c -o python/python__jellyfish_la-swig_wrap.lo `test -f 'python/swig_wrap.cpp' || echo './'`python/swig_wrap.cpp
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I/apps/x86_64/python/2.7.6/include/python2.7 -I../include -std=c++0x -MT python/python__jellyfish_la-swig_wrap.lo -MD -MP -MF python/.deps/python__jellyfish_la-swig_wrap.Tpo -c python/swig_wrap.cpp  -fPIC -DPIC -o python/.libs/python__jellyfish_la-swig_wrap.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I/apps/x86_64/python/2.7.6/include/python2.7 -I../include -std=c++0x -MT python/python__jellyfish_la-swig_wrap.lo -MD -MP -MF python/.deps/python__jellyfish_la-swig_wrap.Tpo -c python/swig_wrap.cpp -o python/python__jellyfish_la-swig_wrap.o >/dev/null 2>&1
mv -f python/.deps/python__jellyfish_la-swig_wrap.Tpo python/.deps/python__jellyfish_la-swig_wrap.Plo
/bin/sh ../libtool  --tag=CXX   --mode=link g++  -std=c++0x  -module -L/apps/x86_64/python/2.7.6/lib -lpython2.7  -o python/_jellyfish.la -rpath /apps/x86_64/python/2.7.6/lib/python2.7/site-packages/jellyfish python/python__jellyfish_la-swig_wrap.lo  ../libjellyfish-2.0.la 
libtool: link: g++ -shared -nostdlib /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/crti.o /usr/lib/gcc/x86_64-redhat-linux/4.4.7/crtbeginS.o  python/.libs/python__jellyfish_la-swig_wrap.o   -Wl,-rpath -Wl,/scicomp/home/xzy3/builds/Jellyfish/.libs -Wl,-rpath -Wl,/scicomp/home/xzy3/.local/lib -L/apps/x86_64/python/2.7.6/lib -lpython2.7 ../.libs/libjellyfish-2.0.so -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-redhat-linux/4.4.7/crtendS.o /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/crtn.o    -Wl,-soname -Wl,_jellyfish.so.0 -o python/.libs/_jellyfish.so.0.0.0
/usr/bin/ld: /apps/x86_64/python/2.7.6/lib/libpython2.7.a(abstract.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC
/apps/x86_64/python/2.7.6/lib/libpython2.7.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make[1]: *** [python/_jellyfish.la] Error 1
make[1]: Leaving directory `/scicomp/home/xzy3/builds/Jellyfish/swig'
make: *** [all] Error 2
gmarcais commented 8 years ago

That does not look right. The flags -L/apps/x86_64/python/2.7.6/lib -lpython2.7 should not be there. Are you sure you pulled in the latest 'develop' branch? Commit 9fe8a3b2eb785d56c1bd5483f9499b49612ca732.

Maybe you need to do:

make distclean
./configure --enable-swig --enable-python-binding --prefix=$(readlink -f ~/.local)
make
xzy3 commented 8 years ago

My fault I was on the wrong branch.

The build works now. The install is trying to put something in the wrong prefix though:

test -z "/apps/x86_64/python/2.7.6/lib/python2.7/site-packages/jellyfish" || /usr/local/bin/mkdir -p "/apps/x86_64/python/2.7.6/lib/python2.7/site-packages/jellyfish"
/usr/local/bin/mkdir: cannot create directory `/apps/x86_64/python/2.7.6/lib/python2.7/site-packages/jellyfish': Permission denied
make[4]: *** [install-pythonextLTLIBRARIES] Error 1

Should be ~/.local/lib/python2.7/site-packages/jellyfish

gmarcais commented 8 years ago

That was actually the intended behavior. --prefix applied to the Jellyfish library per say. You could specify a place to install the binding by doing, for example, --enable-python-binding=$(libdir)/python.

I agree though, it is counter intuitive. The new behavior is to install within prefix if passed to configure, as you expected. Pull develop (commit b2ae882b0ba27560ba3683b44129437d0a25de61).

xzy3 commented 8 years ago

That's kind of clumsy but it seems to work. Thanks for that, I can now successfully build, install, and import jellyfish to non standard locations!