building python3 dynamic libraries as static on linux

[ghost]

Originally reported by: genomematt (Bitbucket: genomematt, GitHub: genomematt)

---

setuptools tries to use the non existent dl to determine whether to build a static or dynamic library resulting in always building static libraries.

Eventually results in an OSerror "cannot open shared object file: No such file or directory"

ugly work around for anyone who got here via googling. At top of setup.py add:

#!python

import sys, types
try:
  import dl
except ImportError:
  sys.modules['dl'] = types.ModuleType('dl')
  sys.modules['dl'].RTLD_NOW = None

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/385

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I confirmed this behavior by forking the repo, putting a branch on the parent commit, checking out that branch, installing the package with python3 setup.py install --user and then invoking ~/.local/bin/ambivert. Here's the traceback.

Traceback (most recent call last):
  File "/home/vagrant/.local/bin/ambivert", line 9, in <module>
    load_entry_point('ambivert==0.5b1', 'console_scripts', 'ambivert')()
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 552, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 2672, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 2345, in load
    return self.resolve()
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 2351, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/vagrant/.local/lib/python3.4/site-packages/ambivert-0.5b1-py3.4-linux-x86_64.egg/ambivert/ambivert.py", line 55, in <module>
    import ambivert.align
  File "/home/vagrant/.local/lib/python3.4/site-packages/ambivert-0.5b1-py3.4-linux-x86_64.egg/ambivert/align/__init__.py", line 1, in <module>
    from ambivert.align.align_ctypes import *
  File "/home/vagrant/.local/lib/python3.4/site-packages/ambivert-0.5b1-py3.4-linux-x86_64.egg/ambivert/align/align_ctypes.py", line 38, in <module>
    output_dir = os.path.split(__file__)[0]))
  File "/usr/lib/python3.4/ctypes/__init__.py", line 429, in LoadLibrary
    return self._dlltype(name)
  File "/usr/lib/python3.4/ctypes/__init__.py", line 351, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/vagrant/.local/lib/python3.4/site-packages/ambivert-0.5b1-py3.4-linux-x86_64.egg/ambivert/align/libalign_c.so: cannot open shared object file: No such file or directory

[ghost]

Original comment by folded (Bitbucket: folded, GitHub: folded):

---

What's happening here is that build_ext is deciding always to build a static library on python3 (because the dl module no longer exists, and os != 'nt'). We expect a shared library to be built by ext_modules = [ Library(...) ], and when we go and use distutils.core to work out the output file (we've lost the name of the file that build_ext actually produced) to load with ctypes, we throw an OSError because that file does not exist.

The hack above just creates a fake dl module so that setuptools always compiles a shared library. It's not a great solution because we don't need a dl- stub python file, but it gets around the problem.

We basically just need a way to force setuptools to build a shared library with a given (or inferrable) filename. The nicest solution would be if Library had a SharedLibrary subclass that always compiled a shared library with a given base, and a platform appropriate extension, that would then be loadable by ctypes.

[ghost]

Original comment by genomematt (Bitbucket: genomematt, GitHub: genomematt):

---

@jaraco Note that the failure is completely silent in the build. It 'succeeds' but just builds the wrong type of library. The OS error is on the attempt to load when you run the program and there is no library with the correct name

[ghost]

Original comment by genomematt (Bitbucket: genomematt, GitHub: genomematt):

---

The project I am working on is https://github.com/genomematt/AmBiVErT

My unix is a clean vagrant default 14.04 with apt-get install python3-pip followed by pip upgrading pip and setuptools

Was working on MacOS but not unix vm or travis prior to https://github.com/genomematt/AmBiVErT/commit/b6f01b508f387ee4ec14e319b277def746c82ae7

needed complementary patch

https://github.com/genomematt/AmBiVErT/commit/dcae55dd249f4adf7fbf6cd554ae665649fc5764

after which it works on unix vm and docker container.

Also using TravisCI which is some of the recent git tiggering. Current version works VM but not Travis where the python2 virtualenv is not compatible with the path hack as os.path.split(file)[0] is not the correct directory

cc @folded

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Here is where build_ext tests for the dl module. That code has a lot of different code paths for different platforms and architectures.

Based on Arfrever's indication, and the code, I would not expect an OS error anywhere in the code. I would expect the import dl call to fail with an ImportError in every case. The fact that it doesn't suggests there may be an improper build on the system.

@genomematt Can you provide more detail that would demonstrate how to replicate the failure?

[ghost]

Original comment by arfrever (Bitbucket: arfrever, GitHub: arfrever):

---

dl module exists only in Python 2 and only in 32-bit architectures.

Fragment of setup.py of Python 2.7:

        # Dynamic loading module
        if sys.maxint == 0x7fffffff:
            # This requires sizeof(int) == sizeof(long) == sizeof(char*)
            dl_inc = find_file('dlfcn.h', [], inc_dirs)
            if (dl_inc is not None) and (host_platform not in ['atheos']):
                exts.append( Extension('dl', ['dlmodule.c']) )
            else:
                missing.append('dl')
        else:
            missing.append('dl')

[abravalheri]

It seems that the RTLD_NOW flag is defined in the os module for Python3: https://docs.python.org/3/library/os.html#os.RTLD_NOW.

Would it be safe if we simply replace the checks from dl to os and ctypes when appropriate?