v0.14.1 fails to import on windows due to pip's isolated build environment

[carterbox]

Issue: Import fails for skimage=0.14.1 when trying to import tifffile when numpy=1.13. However, it doesn't fail when skimage=0.14.0. Dependencies should stay the same for minor revisions(?), so skimage=0.14.1 should also be built against numpy=1.13. Alternatively, if skimage isn't being built against numpy<0.14 there should be a runtime requirement such that conda doesn't install a numpy version that is too low.

Installation of numpy from the defaults channels is always possible because skimage has dependencies in the defaults channel, and it may happen (tomopy/tomopy#364) that other packages in the environment require a numpy version that is not available on conda-forge.

import skimage
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\dching\AppData\Local\Continuum\miniconda3\envs\skimage\lib\site-packages\skimage\__init__.py", line 177, in <module>
    from .data import data_dir
  File "C:\Users\dching\AppData\Local\Continuum\miniconda3\envs\skimage\lib\site-packages\skimage\data\__init__.py", line 15, in <module>
    from ..io import imread, use_plugin
  File "C:\Users\dching\AppData\Local\Continuum\miniconda3\envs\skimage\lib\site-packages\skimage\io\__init__.py", line 7, in <module>
    from .manage_plugins import *
  File "C:\Users\dching\AppData\Local\Continuum\miniconda3\envs\skimage\lib\site-packages\skimage\io\manage_plugins.py", line 28, in <module>
    from .collection import imread_collection_wrapper
  File "C:\Users\dching\AppData\Local\Continuum\miniconda3\envs\skimage\lib\site-packages\skimage\io\collection.py", line 14, in <module>
    from ..external.tifffile import TiffFile
  File "C:\Users\dching\AppData\Local\Continuum\miniconda3\envs\skimage\lib\site-packages\skimage\external\tifffile\__init__.py", line 1, in <module>
    from .tifffile import imsave, imread, imshow, TiffFile, TiffWriter, TiffSequence
  File "C:\Users\dching\AppData\Local\Continuum\miniconda3\envs\skimage\lib\site-packages\skimage\external\tifffile\tifffile.py", line 293, in <module>
    from . import _tifffile
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb

Environment (conda list):

``` $ conda list # Name Version Build Channel blas 1.0 mkl ca-certificates 2018.10.15 ha4d7672_0 conda-forge certifi 2018.10.15 py36_1000 conda-forge cloudpickle 0.6.1 py_0 conda-forge cycler 0.10.0 py_1 conda-forge cytoolz 0.9.0.1 py36hfa6e2cd_1001 conda-forge dask-core 0.20.1 py_0 conda-forge decorator 4.3.0 py_0 conda-forge freetype 2.8.1 vc14_0 [vc14] conda-forge icc_rt 2017.0.4 h97af966_0 icu 58.2 vc14_0 [vc14] conda-forge imageio 2.4.1 py36_1000 conda-forge intel-openmp 2019.0 118 jpeg 9b vc14_2 [vc14] conda-forge kiwisolver 1.0.1 py36he980bc4_1002 conda-forge libpng 1.6.34 vc14_0 [vc14] conda-forge libtiff 4.0.9 vc14_0 [vc14] conda-forge matplotlib 2.2.2 py36_1 conda-forge mkl 2019.0 118 mkl_fft 1.0.6 py36_0 conda-forge mkl_random 1.0.2 py36_0 conda-forge networkx 2.2 py_1 conda-forge numpy 1.13.3 py36h5c71026_4 numpy-base 1.15.4 py36h8128ebf_0 olefile 0.46 py_0 conda-forge openssl 1.0.2p hfa6e2cd_1001 conda-forge pillow 5.2.0 py36_0 conda-forge pip 18.1 py36_1000 conda-forge pyparsing 2.3.0 py_0 conda-forge pyqt 5.6.0 py36h764d66f_7 conda-forge python 3.6.6 he025d50_0 conda-forge python-dateutil 2.7.5 py_0 conda-forge pytz 2018.7 py_0 conda-forge pywavelets 1.0.1 py36h452e1ab_1000 conda-forge qt 5.6.2 vc14_1 [vc14] conda-forge scikit-image 0.14.1 py36h6538335_1001 conda-forge scipy 1.1.0 py36hc28095f_0 setuptools 40.6.2 py36_0 conda-forge sip 4.18.1 py36h6538335_0 conda-forge six 1.11.0 py36_1001 conda-forge tk 8.6.8 vc14_0 [vc14] conda-forge toolz 0.9.0 py_1 conda-forge tornado 5.1.1 py36hfa6e2cd_1000 conda-forge vc 14 0 conda-forge vs2015_runtime 14.0.25420 0 conda-forge wheel 0.32.2 py36_0 conda-forge wincertstore 0.2 py36_1002 conda-forge zlib 1.2.11 vc14_0 [vc14] conda-forge ```

Details about conda and system ( conda info ):

``` $ conda info active environment : skimage active env location : C:\Users\dching\AppData\Local\Continuum\miniconda3\envs\skimage shell level : 2 user config file : C:\Users\dching\.condarc populated config files : C:\Users\dching\.condarc conda version : 4.5.11 conda-build version : 3.16.1 python version : 3.6.6.final.0 base environment : C:\Users\dching\AppData\Local\Continuum\miniconda3 (writable) channel URLs : https://conda.anaconda.org/astra-toolbox/win-64 https://conda.anaconda.org/astra-toolbox/noarch https://conda.anaconda.org/conda-forge/win-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/win-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/free/win-64 https://repo.anaconda.com/pkgs/free/noarch https://repo.anaconda.com/pkgs/r/win-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/pro/win-64 https://repo.anaconda.com/pkgs/pro/noarch https://repo.anaconda.com/pkgs/msys2/win-64 https://repo.anaconda.com/pkgs/msys2/noarch package cache : C:\Users\dching\AppData\Local\Continuum\miniconda3\pkgs C:\Users\dching\AppData\Local\conda\conda\pkgs envs directories : C:\Users\dching\AppData\Local\Continuum\miniconda3\envs C:\Users\dching\AppData\Local\conda\conda\envs C:\Users\dching\.conda\envs platform : win-64 user-agent : conda/4.5.11 requests/2.20.0 CPython/3.6.6 Windows/10 Windows/10.0.17134 administrator : False netrc file : None offline mode : False ```

[hmaarrfk]

I'm worried that you have numpy-base=1.15 installed. Can you try to create a new environment with

conda create --name np13 numpy=1.13 scikit-image=0.14
conda activate np13
python -c 'import skimage; print(skimage.__version__)'

and report back if you have an error.

In linux:

$ python                                                                                                                                                      
Python 3.6.7 | packaged by conda-forge | (default, Nov 21 2018, 02:32:25) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import skimage
>>> skimage.__version__
'0.14.1'
>>> import numpy
>>> numpy.__version__
'1.13.3'

[carterbox]

It's the same error as above.

[hmaarrfk]

@carterbox, have you been able to solve your issue? Pinnings have been updated recently so maybe you shouldn't hit the same issues (or maybe you will).

If the issue still happens, please send me your new conda list in the new env.

Finally you can probably try:

conda create --name np13 numpy=1.13 scikit-image=0.14 "blas=*=openblas"
conda activate np13
python -c 'import skimage; print(skimage.__version__)'

[carterbox]

I still have the same error, and I can't create the suggested environment because of dependency conflicts, so there is no conda list to share.

C:\Users\dchin>conda create --name np13 numpy=1.13 scikit-image=0.14 "blas=*=openblas"
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - blas[build=openblas]
  - numpy=1.13 -> blas=[build=mkl]
Use "conda info <package>" to see the dependencies for each package.

As I suggested initially, if scikit-image no longer works with numpy=1.13 on Windows, there should be a dependency requirement for numpy>1.13 (or something?). Have you been able to recreate this problem?

[hmaarrfk]

We test scikit image with numpy 1.11, but only on linux. Technically it should be ok.

Can you please create a new environment that works for your other dependencies and paste the environment again. I can then get on my windows machine and try to recreate it. If possible, please provie me the commands to recreate the install too.

I'm sorry I'm asking you to do this again, but many updates have happened to numpy and pinnings.

Please do not use the --offline flag.

Grab a coffee for me while you do it ;)

[carterbox]

Here is how to recreate the environment:

conda create --name coffee numpy=1.13 scikit-image=0.14

The result is the following environment. The text below is the yaml from conda env export.

name: coffee
channels:
  - conda-forge
  - defaults
dependencies:
  - ca-certificates=2018.11.29=ha4d7672_0
  - certifi=2018.11.29=py36_1000
  - cloudpickle=0.6.1=py_0
  - cycler=0.10.0=py_1
  - cytoolz=0.9.0.1=py36hfa6e2cd_1001
  - dask-core=1.0.0=py_0
  - decorator=4.3.0=py_0
  - freetype=2.9.1=he8b6a0d_1004
  - icu=58.2=vc14_0
  - imageio=2.4.1=py36_1000
  - jpeg=9c=hfa6e2cd_1001
  - kiwisolver=1.0.1=py36he980bc4_1002
  - libpng=1.6.36=h7602738_1000
  - libtiff=4.0.10=h36446d0_1001
  - matplotlib=2.2.3=py36h31860fd_0
  - mkl_fft=1.0.10=py36_0
  - mkl_random=1.0.2=py36_0
  - networkx=2.2=py_1
  - olefile=0.46=py_0
  - openssl=1.0.2p=hfa6e2cd_1001
  - pillow=5.3.0=py36h9a613e6_1000
  - pip=18.1=py36_1000
  - pyparsing=2.3.0=py_0
  - pyqt=5.6.0=py36h764d66f_1008
  - python=3.6.6=he025d50_0
  - python-dateutil=2.7.5=py_0
  - pytz=2018.7=py_0
  - pywavelets=1.0.1=py36h452e1ab_1000
  - qt=5.6.2=h2639256_8
  - scikit-image=0.14.1=py36h6538335_1001
  - setuptools=40.6.3=py36_0
  - sip=4.18.1=py36h6538335_1000
  - six=1.12.0=py36_1000
  - sqlite=3.26.0=hfa6e2cd_1000
  - tk=8.6.9=hfa6e2cd_1000
  - toolz=0.9.0=py_1
  - tornado=5.1.1=py36hfa6e2cd_1000
  - vc=14=0
  - vs2015_runtime=14.0.25420=0
  - wheel=0.32.3=py36_0
  - wincertstore=0.2=py36_1002
  - zlib=1.2.11=h2fa13f4_1003
  - blas=1.0=mkl
  - icc_rt=2019.0.0=h0cc432a_1
  - intel-openmp=2019.1=144
  - mkl=2018.0.3=1
  - numpy=1.13.3=py36h5c71026_4
  - scipy=1.1.0=py36hc28095f_0
  - pip:
    - dask==1.0.0
prefix: C:\Users\dchin\Miniconda3\envs\coffee

When I import skimage in this environment, then the import error from the original post occurs.

[hmaarrfk]

i was able to recreate it. Thanks!

I have an idea, but it isn't really a good one. let me check.

[hmaarrfk]

whats your workaround?

[hmaarrfk]

@conda-forge/python It seems that pip is creating an isolated build for windows!!!!

I don't know if it is because it has a pyproject.toml https://github.com/scikit-image/scikit-image/blob/v0.14.1/pyproject.toml

See Appveyor build https://ci.appveyor.com/project/conda-forge/scikit-image-feedstock/branch/master/job/6bf93qcd53qq82ht?fullLog=true#L3592

  Installing collected packages: setuptools, wheel, numpy, cython
    Creating C:\Users\appveyor\AppData\Local\Temp\1\pip-build-env-yqqa4jei\Scripts

It doesn't seem to be doing the same for OSX or Linux.

[carterbox]

My workaround has been to install scikit-image=0.14.0. I don't know anything about how scikit-image is built. I only know that v0.14.0 doesn't have the import error, and v0.14.1 does have the error.

[hmaarrfk]

@carterbox thank you very much for following up with this. I think this is a super serious issue.

Basically, when this build passes, I should be able to merge in a fix for you (and everybody not using numpy 1.15!!!!) https://github.com/conda-forge/scikit-image-feedstock/pull/38 Unfortunately, I'm going to have to be a little patient before merging it in.

Your workaround is probably OK if you don't need 0.14.1. If you do need 0.14.1 you can try:

conda remove scikit-image --force
pip install scikit-image --no-deps

CC: @scikit-image/core, @jni maybe we should pin numpy in pyproject.toml?????

[jni]

@hmaarrfk I'm afraid I don't know what an "isolated build" means and how it is causing the problems described here... Can you write a bit more detail about what's happening, why, and whether it's our fault? =)

[hmaarrfk]

build isolation is pip creating a fresh environment just to "build" the src package before instlling it.

it is downloading and using numpy 1.15 for its build on windows.

[hmaarrfk]

@carterbox your specific issue should be fixed in about 1 hour (i.e. when the CIs finish their run and upload the packages).

Python 3.7 is still broken for windows, so I'm going to keep this issue open for now.

[jakirkham]

Can you please raise an issue on the conda-build issue tracker about the pip build isolation issue?

[carterbox]

Thanks, @hmaarrfk!

[hmaarrfk]

Thanks @carterbox. New packages have been uploaded to fix the previous issues.