errors importing native C library (python 3.8.6 & swig 4.0.1)

[tomcucinotta]

Hi, I'm recompiling somoclu on Ubuntu 20.10, with python 3.8.6, and I'm having this problem

• ./autogen.sh
• ./configure
• make => the src/libsomoclu.so library is compiled, and the src/somoclu binary works
• make python; sudo make python_install => all seems fine (see below)

  [...]
  Installed /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg
  [...]

• but when attempting to use it, it fails to import the C native library

  $ python
  Python 3.8.6 (default, May 27 2021, 13:28:02) 
  [GCC 10.2.0] on linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import somoclu
  Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/somoclu/__init__.py", line 12, in <module>
  from .train import Somoclu
  File "/usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/somoclu/train.py", line 25, in <module>
  from .somoclu_wrap import train as wrap_train
  File "/usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/somoclu/somoclu_wrap.py", line 13, in <module>
  from . import _somoclu_wrap
  ImportError: cannot import name '_somoclu_wrap' from partially initialized module 'somoclu' (most likely due to a circular import) (/usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/somoclu/__init__.py)

  the error is highlighted re-raise-ing the exception Any clue ? AFAICR, I wasn't getting into this with python3.6 (if memory serves). Thanks.

$ find /usr/local/lib/python3.8/dist-packages/ | grep somoclu /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/somoclu /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/somoclu/train.py /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/somoclu/init.py /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/somoclu/pycache /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/somoclu/pycache/init.cpython-38.pyc /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/somoclu/pycache/train.cpython-38.pyc /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/somoclu/pycache/somoclu_wrap.cpython-38.pyc /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/somoclu/somoclu_wrap.py /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/_somoclu_wrap.cpython-38-x86_64-linux-gnu.so /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/_somoclu_wrap.py /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/EGG-INFO /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/EGG-INFO/not-zip-safe /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/EGG-INFO/PKG-INFO /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/EGG-INFO/native_libs.txt /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/EGG-INFO/top_level.txt /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/EGG-INFO/dependency_links.txt /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/EGG-INFO/requires.txt /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/EGG-INFO/SOURCES.txt /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/pycache /usr/local/lib/python3.8/dist-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/pycache/_somoclu_wrap.cpython-38.pyc

[tomcucinotta]

tracked down to swig (4.0.1-5build1) generating a somoclu_wrap.py with these lines

# Import the low-level C/C++ module
if __package__ or "." in __name__:
    from . import _somoclu_wrap
else:
    import _somoclu_wrap

if I get rid of the "from ." and just do "import __somoclu_wrap", then I can import somoclu from python.

But this is not an area I'm familiar with, at all!

[tomcucinotta]

Ubuntu 20.10 has a means for downgrading swig:

sudo apt-get purge swig*
sudo apt-get install swig3.0

# this is needed otherwise make python won't find swig
sudo ln -s /usr/bin/swig3.0 /usr/bin/swig

then, after recompiling (make python) and installing (sudo make python_install), it imports correctly.

The code to find the native C library in the same auto-generated somoclu_wrap.py is now a hell of code:

from sys import version_info as _swig_python_version_info
if _swig_python_version_info >= (2, 7, 0):
    def swig_import_helper():
        import importlib
        pkg = __name__.rpartition('.')[0]
        mname = '.'.join((pkg, '_somoclu_wrap')).lstrip('.')
        try:
            return importlib.import_module(mname)
        except ImportError:
            return importlib.import_module('_somoclu_wrap')
    _somoclu_wrap = swig_import_helper()
    del swig_import_helper
elif _swig_python_version_info >= (2, 6, 0):
    def swig_import_helper():
        from os.path import dirname
        import imp
        fp = None
        try:
            fp, pathname, description = imp.find_module('_somoclu_wrap', [dirname(__file__)])
        except ImportError:
            import _somoclu_wrap
            return _somoclu_wrap
        try:
            _mod = imp.load_module('_somoclu_wrap', fp, pathname, description)
        finally:
            if fp is not None:
                fp.close()
        return _mod
    _somoclu_wrap = swig_import_helper()
    del swig_import_helper
else:
    import _somoclu_wrap

Any clue as to how to get this working on the latest swig and avoid this nightmare on recent distros ?

Thanks!

[xgdgsc]

I tested in a anaconda python 3.8 env with swig 4 installed from anaconda in ubuntu 20.04. After building and installing. cd /home/user/anaconda3/envs/p38/lib/python3.8/site-packages/somoclu-1.7.5-py3.8-linux-x86_64.egg/ and cp _somoclu_wrap.cpython-38-x86_64-linux-gnu.so _somoclu_wrap.py somoclu seems to work (only when cd in this directory).

[xgdgsc]

From digging some swig github issues I found this seems to work. You can test.