Crash on install of SimPEG seems to be caused by line in discretize/setup.py

[Rockpointgeo]

Hi all,

Trying to install SimPEG on my mac. Tried using pip in both python 2.7 and 3.5. I got the same error for both.

Collecting discretize (from SimPEG) Using cached discretize-0.1.4.tar.gz Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 1, in File "/private/var/folders/kb/40gm7_gx27178vg4v7gswlxh0000gn/T/pip-build-yni1k5cp/discretize/setup.py", line 9, in from numpy.distutils.core import setup ImportError: No module named 'numpy'

Looking at the setup.py file I see that on line 9 it says

from numpy.distutils.core import setup

By no means a python expert but looking at other setup.py files and they mainly seem to have the line

from distutils.core import setup

Could this be the problem? If not back to the drawing board. Any help would be appreciated.

Thanks Sean

[lheagy]

hi @Rockpointgeo, do you have python installed through anaconda (https://www.continuum.io/downloads)?

It look like python / pypi might not be pathed properly on your machine. If you open a terminal and tun which python what is the output?

[lheagy]

@rowanc1, @jcapriot: do you guys have any thoughts / comments on best practices for creating setup.py files for a package? That is something I could use some pointers on (and in particular when helping troubleshoot instillation)

[jcapriot]

My first guess looking at the error is to make sure numpy is installed properly. The setup.py files do use numpy's version of distutils, but obviously that requires numpy to be installed first. So I'd start by checking that.

The idea here is to follow scipy's setup.py structure, (which uses numpy's distutils on build) as it allows for the modularized versions that make it a little easier to link in external compiled libraries.

[lheagy]

I think we got it sorted for sean's case (I will let him confirm!). Is the main motivation for using the numpy distutils (rather than the core distutils) for cython?

[Rockpointgeo]

Lindsey gave me some help on this. I tried a few things.

1) If I install python using Anaconda during installation many packages are installed including numpy. Then if I install SimPEG via pip there are no errors.

2) If I install python using something else (pyenv via Homebrew for example) no packages are installed during the python installation. If I install SimPEG via pip I get the error.

3) If I install python using something else (pyenv via Homebrew for example) then install the packages listed as dependencies (numpy, scipy, matplotlib, and cython). If I install SimPEG via pip everything is ok.

So this seems to tell me that I need to install numpy before installing SimPEG. But I thought one of nice things pip did was go and grab dependencies for you (could be wrong). I am well out of my depth in terms of python packages etc... So I'll leave it to you guys. The good news is I do have SimPEG installed.

[leouieda]

@lheagy it might be worth passing install_requires=['numpy'] to setup to avoid this problem. Pip will try to install it before installing discretize and simpeg (if there are binary wheels for numpy then it shouldn't be a big deal).

[leouieda]

Sorry, I just saw that you have that already. In that case, then a try except block for importing numpy would be better. You can raise a proper error if it fails telling the user to install numpy.

[lheagy]

thanks @leouieda, that is a good call

[prisae]

I just got bitten by this problem. While it now gives a proper error message (#56), it does not resolve the problem when you depend on pip install -r requirements.txt, and requirements.txt contains

numpy
discretize

It seems to be a simple enough case. Is there no better way / workaround for this issue? Any ideas?

[prisae]

@jcapriot you mentioned the idea is to follow scipy's setup.py.

But this works: requirements.txt:

scipy

pip install -r requirements.txt

while this does not: requirements.txt:

discretize

pip install -r requirements.txt

So we must be missing something... scipy can install numpy, so should discretize.

[prisae]

Not exactly sure, the SciPy setup is quite complex. But before they import from numpy.distutils.core import setup they import from setuptools import setup (https://github.com/scipy/scipy/blob/master/setup.py#L513).

Also, they only add numpy to install_requires if it is not already installed, which might be a good idea.

The important bit, I think, is here: https://github.com/scipy/scipy/blob/master/setup.py#L459

    # Figure out whether to add ``*_requires = ['numpy']``.
    # We don't want to do that unconditionally, because we risk updating
    # an installed numpy which fails too often.  Just if it's not installed, we
    # may give it a try.  See gh-3379.
    try:
        import numpy
    except ImportError:  # We do not have numpy installed
        build_requires = ['numpy>=1.14.5']
    else:
        # If we're building a wheel, assume there already exist numpy wheels
        # for this platform, so it is safe to add numpy to build requirements.
        # See gh-5184.
        build_requires = (['numpy>=1.14.5'] if 'bdist_wheel' in sys.argv[1:]
                          else [])

so if numpy cannot be install it does NOT fail. This is probably the route to go.

[prisae]

Anyway, I'd like to hear input from someone more familiar with the discretize setup and its history than I am... ;-)

[prisae]

Nope, relevant line is probably this: https://github.com/scipy/scipy/blob/master/setup.py#L527

    else:
        # Don't import numpy here - non-build actions are required to succeed
        # without NumPy for example when pip is used to install Scipy when
        # NumPy is not yet present in the system.

[prisae]

I have a workaround for readthedocs:

• Using a .readthedocs.yml, and
• using two requirements-files, the second one beeing, e.g., requirements-rtfd.txt, and only containing one entry, discretize

This is obviously not beautiful, but it works.

Still I think that the current behaviour is a bug and we should try to fix it.

[prisae]

The issue might be related to numpy, we should keep an eye on it. https://github.com/scikit-fmm/scikit-fmm/issues/34

[prisae]

More thing potentially related:

• https://github.com/scikit-image/scikit-image/issues/4369
• https://github.com/numpy/numpy/pull/15194

[leouieda]

My guess is that pip is running setup.py twice: once on download (to check dependencies potentially) and again on install. The numpy setup function is only needed when compiling/building the package, which might explain the if run_build: block in Scipy. So a possible solution is to import from setuptools instead of raising an exception if numpy is not found. That would allow pip to find the dependencies for discretize and install them before building discretize (which it should be smart enough to do since setup_requires is specified.

[leouieda]

Apparently this is a really common problem. PEP 518 can probably solve this (the pyproject.toml spec) https://www.python.org/dev/peps/pep-0518/