search

DigitalSlideArchive / HistomicsTK

A Python toolkit for pathology image analysis algorithms.
https://digitalslidearchive.github.io/HistomicsTK/
Apache License 2.0
380 stars 113 forks source link

Non-editable mode install of HistomicsTK fails in Cython build #362

Closed predicative closed 5 years ago

predicative commented 7 years ago

The following script to set up HistomicsTK fails on current master (07ec26a60bd64666eb4c989b39c60a0a031276f4):

virtualenv venv
git clone https://github.com/DigitalSlideArchive/HistomicsTK --depth=1
venv/bin/pip install scikit-build
venv/bin/pip install ./HistomicsTK

The output from the last install command is at the end of this message. Cython encounters some build error apparently related to NumPy. However, if I add -e to the last install command, it works. I observe the same pattern if I follow the install instructions more closely, like so:

virtualenv venv
git clone https://github.com/girder/large_image --depth=1
venv/bin/pip install ./large_image
git clone https://github.com/DigitalSlideArchive/HistomicsTK --depth=1
venv/bin/pip install --no-cache-dir -r HistomicsTK/requirements.txt
venv/bin/pip install ./HistomicsTK

Attempting to run pip install ./HistomicsTK again after it initially failed, while it still errors, gives a different, CMake-related error. It also seems to fail similarly on the first commit introducing scikit-build, c34b5e9d9c0e78b3cd1f1b78fe379ce53eaa846a (with the caveat that it uses the most up-to-date packages).

The output and error:

Processing ./HistomicsTK
Collecting Cython>=0.25.2 (from histomicstk==0.2.0)
  Using cached Cython-0.26-cp27-cp27mu-manylinux1_x86_64.whl
Collecting nimfa>=1.3.2 (from histomicstk==0.2.0)
Collecting numpy>=1.12.1 (from histomicstk==0.2.0)
  Using cached numpy-1.13.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pandas>=0.19.2 (from histomicstk==0.2.0)
  Using cached pandas-0.20.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting scikit-image>=0.12.3 (from histomicstk==0.2.0)
  Using cached scikit_image-0.13.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting scikit-learn>=0.18.1 (from histomicstk==0.2.0)
  Using cached scikit_learn-0.18.2-cp27-cp27mu-manylinux1_x86_64.whl
Collecting scipy>=0.19.0 (from histomicstk==0.2.0)
  Using cached scipy-0.19.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting dask>=0.14.2 (from histomicstk==0.2.0)
  Using cached dask-0.15.1-py2.py3-none-any.whl
Collecting distributed>=1.16.3 (from histomicstk==0.2.0)
  Using cached distributed-1.18.0-py2.py3-none-any.whl
Requirement already satisfied: scikit-build>=0.6.0 in ./venv/lib/python2.7/site-packages (from histomicstk==0.2.0)
Collecting cmake>=0.6.0 (from histomicstk==0.2.0)
  Using cached cmake-0.8.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pytz>=2011k (from pandas>=0.19.2->histomicstk==0.2.0)
  Using cached pytz-2017.2-py2.py3-none-any.whl
Collecting python-dateutil (from pandas>=0.19.2->histomicstk==0.2.0)
  Using cached python_dateutil-2.6.1-py2.py3-none-any.whl
Collecting matplotlib>=1.3.1 (from scikit-image>=0.12.3->histomicstk==0.2.0)
  Using cached matplotlib-2.0.2-cp27-cp27mu-manylinux1_x86_64.whl
Collecting six>=1.7.3 (from scikit-image>=0.12.3->histomicstk==0.2.0)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting networkx>=1.8 (from scikit-image>=0.12.3->histomicstk==0.2.0)
  Using cached networkx-1.11-py2.py3-none-any.whl
Collecting pillow>=2.1.0 (from scikit-image>=0.12.3->histomicstk==0.2.0)
  Using cached Pillow-4.2.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting PyWavelets>=0.4.0 (from scikit-image>=0.12.3->histomicstk==0.2.0)
  Using cached PyWavelets-0.5.2-cp27-cp27mu-manylinux1_x86_64.whl
Collecting cloudpickle>=0.2.2 (from distributed>=1.16.3->histomicstk==0.2.0)
  Using cached cloudpickle-0.3.1-py2.py3-none-any.whl
Collecting click>=6.6 (from distributed>=1.16.3->histomicstk==0.2.0)
  Using cached click-6.7-py2.py3-none-any.whl
Collecting tornado>=4.4 (from distributed>=1.16.3->histomicstk==0.2.0)
Collecting zict>=0.1.2 (from distributed>=1.16.3->histomicstk==0.2.0)
  Using cached zict-0.1.2-py2.py3-none-any.whl
Collecting singledispatch; python_version < "3.4" (from distributed>=1.16.3->histomicstk==0.2.0)
  Using cached singledispatch-3.4.0.3-py2.py3-none-any.whl
Collecting sortedcontainers (from distributed>=1.16.3->histomicstk==0.2.0)
  Using cached sortedcontainers-1.5.7-py2.py3-none-any.whl
Collecting toolz>=0.7.4 (from distributed>=1.16.3->histomicstk==0.2.0)
Collecting psutil (from distributed>=1.16.3->histomicstk==0.2.0)
Collecting futures; python_version < "3.0" (from distributed>=1.16.3->histomicstk==0.2.0)
  Using cached futures-3.1.1-py2-none-any.whl
Collecting msgpack-python (from distributed>=1.16.3->histomicstk==0.2.0)
Collecting tblib (from distributed>=1.16.3->histomicstk==0.2.0)
  Using cached tblib-1.3.2-py2.py3-none-any.whl
Requirement already satisfied: wheel>=0.29.0 in ./venv/lib/python2.7/site-packages (from scikit-build>=0.6.0->histomicstk==0.2.0)
Requirement already satisfied: setuptools>=28.0.0 in ./venv/lib/python2.7/site-packages (from scikit-build>=0.6.0->histomicstk==0.2.0)
Collecting cycler>=0.10 (from matplotlib>=1.3.1->scikit-image>=0.12.3->histomicstk==0.2.0)
  Using cached cycler-0.10.0-py2.py3-none-any.whl
Collecting subprocess32 (from matplotlib>=1.3.1->scikit-image>=0.12.3->histomicstk==0.2.0)
Collecting functools32 (from matplotlib>=1.3.1->scikit-image>=0.12.3->histomicstk==0.2.0)
Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 (from matplotlib>=1.3.1->scikit-image>=0.12.3->histomicstk==0.2.0)
  Using cached pyparsing-2.2.0-py2.py3-none-any.whl
Collecting decorator>=3.4.0 (from networkx>=1.8->scikit-image>=0.12.3->histomicstk==0.2.0)
  Using cached decorator-4.1.2-py2.py3-none-any.whl
Collecting olefile (from pillow>=2.1.0->scikit-image>=0.12.3->histomicstk==0.2.0)
Collecting certifi (from tornado>=4.4->distributed>=1.16.3->histomicstk==0.2.0)
  Using cached certifi-2017.7.27.1-py2.py3-none-any.whl
Collecting backports-abc>=0.4 (from tornado>=4.4->distributed>=1.16.3->histomicstk==0.2.0)
  Using cached backports_abc-0.5-py2.py3-none-any.whl
Collecting heapdict (from zict>=0.1.2->distributed>=1.16.3->histomicstk==0.2.0)
Installing collected packages: Cython, numpy, scipy, nimfa, pytz, six, python-dateutil, pandas, cycler, subprocess32, functools32, pyparsing, matplotlib, decorator, networkx, olefile, pillow, PyWavelets, scikit-image, scikit-learn, dask, cloudpickle, click, certifi, singledispatch, backports-abc, tornado, heapdict, zict, sortedcontainers, toolz, psutil, futures, msgpack-python, tblib, distributed, cmake, histomicstk
  Running setup.py install for histomicstk: started
    Running setup.py install for histomicstk: finished with status 'error'
    Complete output from command /home/neal/work/problem_htk5/venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-l_5ByX-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-10jRU7-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/neal/work/problem_htk5/venv/include/site/python2.7/histomicstk:
    cmake version 3.9.0

    CMake suite maintained and supported by Kitware (kitware.com/cmake).

    --------------------------------------------------------------------------------
    -- Trying "Ninja" generator
    --------------------------------
    ---------------------------
    ----------------------
    -----------------
    ------------
    -------
    --
    Not searching for unused variables given on the command line.
    CMake Error: CMake was unable to find a build program corresponding to "Ninja".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
    -- Configuring incomplete, errors occurred!
    --
    -------
    ------------
    -----------------
    ----------------------
    ---------------------------
    --------------------------------
    -- Trying "Ninja" generator - failure
    --------------------------------------------------------------------------------

    --------------------------------------------------------------------------------
    -- Trying "Unix Makefiles" generator
    --------------------------------
    ---------------------------
    ----------------------
    -----------------
    ------------
    -------
    --
    Not searching for unused variables given on the command line.
    -- The CXX compiler identification is GNU 6.3.1
    -- Check for working CXX compiler: /usr/bin/c++
    -- Check for working CXX compiler: /usr/bin/c++ -- works
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- The C compiler identification is GNU 6.3.1
    -- Check for working C compiler: /usr/bin/cc
    -- Check for working C compiler: /usr/bin/cc -- works
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Configuring done
    -- Generating done
    -- Build files have been written to: /tmp/pip-l_5ByX-build/_cmake_test_compile/build
    --
    -------
    ------------
    -----------------
    ----------------------
    ---------------------------
    --------------------------------
    -- Trying "Unix Makefiles" generator - success
    --------------------------------------------------------------------------------

    -- The C compiler identification is GNU 6.3.1
    -- The CXX compiler identification is GNU 6.3.1
    -- Check for working C compiler: /usr/bin/cc
    -- Check for working C compiler: /usr/bin/cc -- works
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Check for working CXX compiler: /usr/bin/c++
    -- Check for working CXX compiler: /usr/bin/c++ -- works
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Found PythonInterp: /home/neal/work/problem_htk5/venv/bin/python (found version "2.7.13")
    -- Found PythonLibs: /usr/lib64/libpython2.7.so (found version "2.7.13")
    -- Found Cython: /home/neal/work/problem_htk5/venv/bin/cython
    -- Found NumPy: /usr/include (found version "1.13.1")
    -- Performing Test Weak Link MODULE -> SHARED (gnu_ld_ignore) - Success
    NUMPY_INCLUDE_DIRS:/usr/include
    -- Configuring done
    -- Generating done
    CMake Warning:
      Manually-specified variables were not used by the project:

        SKBUILD

    -- Build files have been written to: /tmp/pip-l_5ByX-build/_skbuild/cmake-build
    [ 16%] Generating CXX source histomicstk/segmentation/label/_trace_object_boundaries_cython.cxx

    Error compiling Cython file:
    ------------------------------------------------------------
    ...
    import numpy as np
    ^
    ------------------------------------------------------------

    /tmp/pip-l_5ByX-build/histomicstk/segmentation/label/_trace_object_boundaries_cython.pyx:1:0: 'pip-l_5ByX-build.histomicstk.segmentation.label._trace_object_boundaries_cython' is not a valid module name
    histomicstk/segmentation/label/CMakeFiles/_trace_object_boundaries_cython.dir/build.make:61: recept voor doel 'histomicstk/segmentation/label/_trace_object_boundaries_cython.cxx' is mislukt
    gmake[2]: *** [histomicstk/segmentation/label/_trace_object_boundaries_cython.cxx] Fout 1
    gmake[2]: *** Verwijderen van bestand 'histomicstk/segmentation/label/_trace_object_boundaries_cython.cxx'
    CMakeFiles/Makefile2:85: recept voor doel 'histomicstk/segmentation/label/CMakeFiles/_trace_object_boundaries_cython.dir/all' is mislukt
    gmake[1]: *** [histomicstk/segmentation/label/CMakeFiles/_trace_object_boundaries_cython.dir/all] Fout 2
    Makefile:129: recept voor doel 'all' is mislukt
    gmake: *** [all] Fout 2
    Traceback (most recent call last):
      File "/home/neal/work/problem_htk5/venv/lib/python2.7/site-packages/skbuild/setuptools_wrap.py", line 409, in setup
        cmkr.make(make_args, env=env)
      File "/home/neal/work/problem_htk5/venv/lib/python2.7/site-packages/skbuild/cmaker.py", line 417, in make
        os.path.abspath(CMAKE_BUILD_DIR)))

    An error occurred while building with CMake.
      Command:
        "cmake" "--build" "." "--target" "install" "--config" "Release" "--"
      Source directory:
        /tmp/pip-l_5ByX-build
      Working directory:
        /tmp/pip-l_5ByX-build/_skbuild/cmake-build
    Please see CMake's output for more information.

    ----------------------------------------
Command "/home/neal/work/problem_htk5/venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-l_5ByX-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-10jRU7-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/neal/work/problem_htk5/venv/include/site/python2.7/histomicstk" failed with error code 1 in /tmp/pip-l_5ByX-build/
predicative commented 7 years ago

I can also reproduce the problem at 76e78ef10f209956c6ddd96471b872db6be797c4, prior to using scikit-build. However, if I run pip install -e ., I can then run pip install . without problems, unlike what we find with the newest commits.

Processing /home/neal/work/problem_htk/HistomicsTK
    Complete output from command python setup.py egg_info:

    Error compiling Cython file:
    ------------------------------------------------------------
    ...
    import numpy as np
    ^
    ------------------------------------------------------------

    histomicstk/segmentation/label/_trace_object_boundaries_cython.pyx:1:0: 'pip-Ew3mV8-build.histomicstk.segmentation.label._trace_object_boundaries_cython' is not a valid module name
    Warning: Extension name 'histomicstk.segmentation.label._trace_object_boundaries_cython' does not match fully qualified name 'pip-Ew3mV8-build.histomicstk.segmentation.label._trace_object_boundaries_cython' of 'histomicstk/segmentation/label/_trace_object_boundaries_cython.pyx'
    Warning: Extension name 'histomicstk.segmentation.nuclear._max_clustering_cython' does not match fully qualified name 'pip-Ew3mV8-build.histomicstk.segmentation.nuclear._max_clustering_cython' of 'histomicstk/segmentation/nuclear/_max_clustering_cython.pyx'
    Compiling histomicstk/segmentation/label/_trace_object_boundaries_cython.pyx because it changed.
    Compiling histomicstk/segmentation/nuclear/_max_clustering_cython.pyx because it changed.
    [1/2] Cythonizing histomicstk/segmentation/label/_trace_object_boundaries_cython.pyx
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-Ew3mV8-build/setup.py", line 88, in <module>
        ext_modules = cythonize(ext_list)
      File "/home/neal/work/problem_htk/venv/lib/python2.7/site-packages/Cython/Build/Dependencies.py", line 1016, in cythonize
        cythonize_one(*args)
      File "/home/neal/work/problem_htk/venv/lib/python2.7/site-packages/Cython/Build/Dependencies.py", line 1138, in cythonize_one
        raise CompileError(None, pyx_file)
    Cython.Compiler.Errors.CompileError: histomicstk/segmentation/label/_trace_object_boundaries_cython.pyx

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-Ew3mV8-build/
bohu615 commented 7 years ago

please see https://github.com/DigitalSlideArchive/HistomicsTK/issues/346

https://stackoverflow.com/questions/42102656/cython-compile-error-is-not-a-valid-module-name according to this, I delete init.py then it's installed successfully.