Importing package creates a SystemError

[abeschneider]

I installed my_package and ran test.py under extension-ffi/package/test and got the error:

Traceback (most recent call last):
    File "test/test.py", line 4, in <module>
       from my_package.modules.add import MyAddModule
    File "/usr/local/lib/python2.7/site-packages/my_package-0.1-py2.7-macosx-10.12-x86_64.egg/my_package/modules/add.py", line 2, in <module>
       from ..functions.add import MyAddFunction
    File "/usr/local/lib/python2.7/site-packages/my_package-0.1-py2.7-macosx-10.12-x86_64.egg/my_package/functions/add.py", line 4, in <module>
       from .._ext import my_lib
     File "/usr/local/lib/python2.7/site-packages/my_package-0.1-py2.7-macosx-10.12-x86_64.egg/my_package/_ext/my_lib/__init__.py", line 3, in <module>
     from ._my_lib import lib as _lib, ffi as _ffi
SystemError: dynamic module not initialized properly

This was run on Mac OS 10.12.3 using Python 2.7 with the most up to date version of PyTorch from master.

Trying investigate, I tried to import my_package.modules, I don't get an error, but if I try to import my_package.modules.add I get:

ImportError: No module named _my_lib

[thuyen]

Any solution for this? My script is working but the package doesn't (with the same above SystemError). I copied the binary from the compiled script to overwrite binary in the compiled package and it works. How are they (the binaries) different?

[gpleiss]

After some investingation, I think I've pinpointed the problem. It only affects python 2, not python 3 (from tests with anaconda 2.7 and 3.6).

With Python 2, cfft expects the package to be loaded from the directory of build.py. Therefore, if you build the package, and change directories into my_package and run

import _ext.my_lib

then there will be no error. However, running

import my_package._ext.mylib

from the package root directory (or any other directory after installation) will cause the SystemError.

One workaround that I have found is 1) moving the build.py script to the root directory 2) Adding the my_package/ prefix to the headers and source files in build.py 3) Setting ext_package to '' in setup.py. 4) Removing the prefix my_package from lines in setup.py.

This fix makes packages work in Python 2.7, and they still work in Python 3.

I'm not sure if this is a change that should be made to this example code, or if this is a temporary workaround until we can get a bug fixed in cfft. Maintainers, any thoughts?

[apaszke]

@gpleiss Can you please submit a PR? Thank you!

[gpleiss]

Sure thing!