pip3 install . step doesn't install nighresjava properly

[owainkenwayucl]

Hi,

I maintain packages on our compute clusters and one of our researchers has requested nighres. Following the install instructions (for release 1.1.0b) results in an install of nighres which does not have nighresjava after the pip install command is completed.

i.e. I do (within a script (https://github.com/UCL-RITS/rcps-buildscripts/blob/master/nighres-1.1.0b_install)):

pip3 install jcc

# Build nighres
./build.sh

# Install it with pip
pip3 install .

When I run python3 and do import nighres I get the following:

>>> import nighres
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/shared/ucl/apps/nighres/1.1.0b/lib/python3.6/site-packages/nighres/__init__.py", line 2, in <module>
    import nighres.brain
  File "/shared/ucl/apps/nighres/1.1.0b/lib/python3.6/site-packages/nighres/brain/__init__.py", line 1, in <module>
    from nighres.brain.extract_brain_region import extract_brain_region
  File "/shared/ucl/apps/nighres/1.1.0b/lib/python3.6/site-packages/nighres/brain/extract_brain_region.py", line 5, in <module>
    import nighresjava
  File "/shared/ucl/apps/nighres/1.1.0b/lib/python3.6/site-packages/nighresjava/__init__.py", line 3, in <module>
    from . import _nighresjava
ImportError: cannot import name '_nighresjava'

Looking in /shared/ucl/apps/nighres/1.1.0b/lib/python3.6/site-packages/nighresjava/ the only file is the __init__.py (and the pycache). All the .so files etc. are missing.

Manually copying them from the build folder resolves this issue.

Have I gone wrong or have I uncovered a bug?

[juhuntenburg]

Hi @owainkenwayucl, can you make sure pip3 points to the same Python installation that you then use to call nighres (and get the error)? pip3 can be a bit of a mess if you have several python installations and virtual environments, I would recommend directly using python3 -m pip instead. Let me know if that helps, otherwise I will look into it more closely!

[owainkenwayucl]

I believe it is, but just let me do some test builds to double-check!

[owainkenwayucl]

OK,

With pip3:

At the install step, which pip3 gives:

/home/ccspapp/Software/nighres/install/bin/pip3

(ccspapp is a role account we use for software installs`)

Then logged in as ccspapp, looking at that virtualenv:

[ccspapp@login01 ~]$ cd Software/nighres/
[ccspapp@login01 nighres]$ source install/bin/activate
(install) [ccspapp@login01 nighres]$ which python3
~/Software/nighres/install/bin/python3
(install) [ccspapp@login01 nighres]$ python3
Python 3.6.3 (default, Oct 10 2017, 15:08:16) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import nighres
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ccspapp/Software/nighres/install/lib/python3.6/site-packages/nighres/__init__.py", line 2, in <module>
    import nighres.brain
  File "/home/ccspapp/Software/nighres/install/lib/python3.6/site-packages/nighres/brain/__init__.py", line 1, in <module>
    from nighres.brain.extract_brain_region import extract_brain_region
  File "/home/ccspapp/Software/nighres/install/lib/python3.6/site-packages/nighres/brain/extract_brain_region.py", line 5, in <module>
    import nighresjava
  File "/home/ccspapp/Software/nighres/install/lib/python3.6/site-packages/nighresjava/__init__.py", line 3, in <module>
    from . import _nighresjava
ImportError: cannot import name '_nighresjava'
>>> 

The difference between outputs of /home/ccspapp/Software/nighres/install/bin/pip3 in the script and ~/Software/nighres/install/bin/python3 interactively is due to the RHEL default alias which changes the behaviour of which.

(install) [ccspapp@login01 nighres]$ /usr/bin/which python3
/home/ccspapp/Software/nighres/install/bin/python3

I've done a second install, to a second virtualenv with python3 -m pip instead of pip3

[ccspapp@login01 ~]$ cd Software/nighres/
[ccspapp@login01 nighres]$ source install-p/bin/activate
(install-p) [ccspapp@login01 nighres]$ python3
Python 3.6.3 (default, Oct 10 2017, 15:08:16) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import nighres
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ccspapp/Software/nighres/install-p/lib/python3.6/site-packages/nighres/__init__.py", line 2, in <module>
    import nighres.brain
  File "/home/ccspapp/Software/nighres/install-p/lib/python3.6/site-packages/nighres/brain/__init__.py", line 1, in <module>
    from nighres.brain.extract_brain_region import extract_brain_region
  File "/home/ccspapp/Software/nighres/install-p/lib/python3.6/site-packages/nighres/brain/extract_brain_region.py", line 5, in <module>
    import nighresjava
  File "/home/ccspapp/Software/nighres/install-p/lib/python3.6/site-packages/nighresjava/__init__.py", line 3, in <module>
    from . import _nighresjava
ImportError: cannot import name '_nighresjava'
>>> 

[owainkenwayucl]

Oh, if it matters, we are using pip 18.1.

[juhuntenburg]

Yep, I get the same error. I am confused why this didn't happen in any of the testing. @piloubazin any idea what the problem is?

[juhuntenburg]

Ah, I think I might have it, just testing

[juhuntenburg]

Ok, should be fixed on master. The issue was that those files weren't included in the python packaged through MANIFEST.in (see commit)

I will make a new release that fixes this. Thanks for pointing it out @owainkenwayucl !

[owainkenwayucl]

No problem :)

[saurabhpre]

This issue is seen again in version 1.4.0 with python 3.8

[ortiztudela]

Also with python 3.9 and the latest version of nighres. I'm currently trying to install it on a MacBook M2 (in case it is a compatibility issue with the new chips).

[piloubazin]

have you checked that the MANIFEST.in is correct? did the rest of the installation work out for you? New chips shouldn't be a direct issue, but might be a problem for JCC that does the Java -> C translation.