ImportError: No module named pyprf.analysis.__main__

[MSchnei]

After a fresh install and following the installation descriptions in the readme, I tried to run pyprf and got the following error message:

pyprf -config '/media/sf_D_DRIVE/MotDepPrf/Analysis/S01/02_MotLoc/config_S01.csv' 
Traceback (most recent call last):
  File "/home/marian/miniconda2/envs/env4pyprf/bin/pyprf", line 11, in <module>
    load_entry_point('pyprf==1.1.0', 'console_scripts', 'pyprf')()
  File "/home/marian/miniconda2/envs/env4pyprf/lib/python2.7/site-packages/pkg_resources/__init__.py", line 572, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/marian/miniconda2/envs/env4pyprf/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2769, in load_entry_point
    return ep.load()
  File "/home/marian/miniconda2/envs/env4pyprf/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2422, in load
    return self.resolve()
  File "/home/marian/miniconda2/envs/env4pyprf/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2428, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
ImportError: No module named pyprf.analysis.__main__

[ingo-m]

Sorry I am unable to reproduce this error. Does this also occur if you cd into the parent pyprf directory (i.e. cd ~/GitHub/pyprf), and run pytest?

The strange thing is that the current version runs the tests without error locally on debian and mac os, and remotely on travis-ci. The test covers almost all of the code, including model creation.

Also, I pulled and installed from github, and ran the analysis manually (i.e. not using pytest, but from commandline with pyprf -config ~/config.csv).

I suppose the error occured on debian 8 with conda?

Can you check whether you have the following file: ~/pyprf/pyprf/analysis/__main__.py?

[ingo-m]

Can you try

pyprf -config /media/sf_D_DRIVE/MotDepPrf/Analysis/S01/02_MotLoc/config_S01.csv

instead of

pyprf -config '/media/sf_D_DRIVE/MotDepPrf/Analysis/S01/02_MotLoc/config_S01.csv'

?

I.e. without '?

[ingo-m]

Ok so it turns out that pip install -e /path/to/pyprf solves the problem (i.e. using the editable mode). Not sure why though. Any ideas?

[ingo-m]

Probably the problem is related to some remaining relative paths, they need to be updated to absolute (see this post).

[ingo-m]

Although I didn't find any relative import... @ofgulban, do you have any ideas on this?

[ingo-m]

The problem may be related to the version parameter. I found a difference in the following file, depending on whether installation is in editable mode or not (i.e. with or without the -e flag):

The file ~/miniconda2/envs/env4pyprf/bin/ when installing in editable mode:

#!/home/john/miniconda2/envs/py_devel/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'pyprf','console_scripts','pyprf'
__requires__ = 'pyprf'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('pyprf', 'console_scripts', 'pyprf')()
    )

Same file when installing in normal mode:

#!/home/john/miniconda2/envs/env4pyprf/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'pyprf==1.1.0','console_scripts','pyprf'
__requires__ = 'pyprf==1.1.0'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('pyprf==1.1.0', 'console_scripts', 'pyprf')()
    )

@ofgulban , do you think the content of the __init__.py files may play a role?

[ofgulban]

I am working on this atm. The main problem seems to be the relative imports. This is why it cannot find cython module because it does not import from the installed package (with pip install .)

[ofgulban]

ok, I have changed all the relative imports to package imports and made adjustments/simplifications to cytonize the pyx file. Both python setup.py install and python setup.py develop should work now.

I will make a PR but you should test the new instructions etc. before merging.

[ingo-m]

Ok thanks, great work

[MSchnei]

Thanks for looking into this!

[ingo-m]

I will perform more tests and integrate the changes next week.

[ingo-m]

This issue is quite pervasive.

I would like pyprf to be able to be installed with pip install. The reason for this is that I would like to put pyprf on PyPI in the long run.

Now, the current version ofsetup.py on branch faruk_changes starts like this:

from setuptools import setup, Extension
from Cython.Build import cythonize
import numpy

The problem is that the cython and numpy dependencies are imported before any dependencies are installed. In a 'clean' (i.e. empty) virtual environment this results in an error (as expected).

The following solution may be a way to get around the cython dependency in setup.py:

from setuptools import setup, Extension

# Cython dependency removed:
# from Cython.Build import cythonize

# Remaining problem - numpy dependency:
import numpy

# The following part goes into the `ext_modules=...` parameter of `setup`:
# extensions = [Extension("pyprf.analysis.cython_leastsquares",
#                        ["pyprf/analysis/cython_leastsquares.pyx"],
#                        include_dirs=[numpy.get_include()])]

with open('README.md') as f:
    long_description = f.read()

setup(name='pyprf',
      version='1.1.1',
      description=('A free & open source python tool for population receptive \
                    field analysis of fMRI data.'),
      url='https://github.com/ingo-m/pyprf',
      author='Ingo Marquardt',
      license='GNU General Public License Version 3',
      packages=['pyprf.analysis'],
      install_requires=['numpy', 'scipy', 'nibabel', 'pillow', 'cython',
                        'tensorflow'],
      keywords=['pRF', 'fMRI', 'retinotopy'],
      long_description=long_description,
      setup_requires=['cython', 'numpy'],
      # Setuptools properly handles Cython extensions, according to this:
      # https://stackoverflow.com/a/38057196
      ext_modules=[Extension('pyprf.analysis.cython_leastsquares',
                             ['pyprf/analysis/cython_leastsquares.pyx'],
                             include_dirs=[numpy.get_include()]
                             )],
      entry_points={
          'console_scripts': [
              'pyprf = pyprf.analysis.__main__:main',
              ]},
      )

However, we are still left with the numpy dependency at the beginning of the setup.py file. This may be a solution: https://stackoverflow.com/a/21621689 But is it a good solution?

[ofgulban]

I have put the following line in setup.py for this reason: setup_requires=['cython', 'numpy'],

[ofgulban]

Found a more relevant issue: https://github.com/pypa/pip/issues/2381

[ingo-m]

This is the error I get with the current version of branch faruk_changes.

pip install .
Processing /Users/john/1_PhD/GitHub/pyprf
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/vx/0hpwp2_s46j29m_wsbynw3vm0000gn/T/pip-gC2Kri-build/setup.py", line 9, in <module>
        from Cython.Build import cythonize
    ImportError: No module named Cython.Build

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/vx/0hpwp2_s46j29m_wsbynw3vm0000gn/T/pip-gC2Kri-build/

If I use the modified setup.py file (see my previous post, where I don't use from Cython.Build import cythonize but instead let setuptools handle the cython building), the same thing happens with the other dependency, i.e. numpy:

pip install .
Processing /Users/john/1_PhD/GitHub/pyprf
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/vx/0hpwp2_s46j29m_wsbynw3vm0000gn/T/pip-U5z4HI-build/setup.py", line 14, in <module>
        import numpy
    ImportError: No module named numpy

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/vx/0hpwp2_s46j29m_wsbynw3vm0000gn/T/pip-U5z4HI-build/

In short: When using pip install, the two imports at the beginning of the setup.py file (from Cython.Build import cythonize and import numpy) are not possible. My previous posts solves the cython dependency, and the stackoverflow solution may solve the numpy dependency (I will try that later). But it doesn't seem "elegant" :)

[ingo-m]

This issue was very persistent but I think it has now been solved.

In summary, the following aspects played a role in this issue:

• It should be possible to install pyprf from pypi (simply using pip install pyprf).
• There is cython code that needs to be complied for pyprf to work.
• Tests should work both locally during development (using pytest locally) and remotely after committing (on travis-ci.org).

In order for pyprf to work, compiled cython code needs to be available in the installation directory (e.g. at ~/miniconda2/envs/my_env/lib/python2.7/site-packages/pyprf/analysis/cython_leastsquares.so). In order for local tests to work, compiled cython code also needs to be available in the working (git) directory. This latter requirement was always fulfilled (this is why pip install -e /path/to/pyprf and python setup.py develop worked, whereas pip install /path/to/pyprf did not work; the compiled cython code was available in the working directory, but not in the installation directory. In order to solve this issue and have the tests work both locally and remotely, some redundancy was necessary (in order to set up the compiled cython code for both cases).

Moreover, in order to compile cython during setup, numpy is necessary, so numpy needs to be installed before pip install pyprf.

The following commits are related to solving this issue:

• b77ba2bc300a4b20260adc5baabd6932f97a6009
• 3c6fb92779b7d1cc6d4e246e0d72c1492c13a474
• fd65e3b5a656146711d9358e9f755a850992011a
• 99cbe0210eb8c9e3ad843e428ded4cee1767b68b

Future improvements may result in a more elegant implementation (e.g. getting rid of the numpy dependency for setup), but for now the issue is solved.