search

PDAL / python

PDAL's Python Support
Other
117 stars 35 forks source link

Broken pip install #79

Closed gsakkis closed 3 years ago

gsakkis commented 3 years ago

Installing via pip seems to succeed but the installation is incomplete:

$ pip3 install pdal
Collecting pdal
  Downloading PDAL-2.3.6.tar.gz (942 kB)
     |████████████████████████████████| 942 kB 4.5 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: pdal
  Building wheel for pdal (PEP 517) ... done
  Created wheel for pdal: filename=PDAL-2.3.6-py3-none-any.whl size=6467 sha256=7516dac52b0808a3e075a7f91c8de5ad633cfc51c334100d9c4be4334d3f5da4
  Stored in directory: /home/ubuntu/.cache/pip/wheels/12/ef/4f/30f6a1126d2a413b49a6208034bf1262b219b848ebd3025ebd
Successfully built pdal
Installing collected packages: pdal
Successfully installed pdal-2.3.6

$ python -c "import pdal"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pdal/__init__.py", line 3, in <module>
    from .pipeline import Pipeline
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pdal/pipeline.py", line 2, in <module>
    from pdal import libpdalpython
ImportError: cannot import name 'libpdalpython' from partially initialized module 'pdal' (most likely due to a circular import) (/home/ubuntu/.local/lib/python3.8/site-packages/pdal/__init__.py)
hobu commented 3 years ago

Is the PDAL base library installed? It isn't included in the Python PDAL wheel (which is just the extension and PDAL filters.python).

gsakkis commented 3 years ago

Yes it is (built from source):

 pdal --version
-----------------------------------------------------------------------------------------------------------------------------------
pdal 2.2.0 (git-version: Release)
-----------------------------------------------------------------------------------------------------------------------------------

I managed to build pdal-python manually from source after installing scikit-build:

$ python setup.py build
Traceback (most recent call last):
  File "setup.py", line 22, in <module>
    from skbuild import setup
ModuleNotFoundError: No module named 'skbuild'

$ pip3 install scikit-build
Collecting scikit-build
  Downloading scikit_build-0.11.1-py2.py3-none-any.whl (72 kB)
     |████████████████████████████████| 72 kB 1.5 MB/s 
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from scikit-build) (20.7)
Requirement already satisfied: wheel>=0.29.0 in /usr/lib/python3/dist-packages (from scikit-build) (0.34.2)
Requirement already satisfied: setuptools>=28.0.0; python_version >= "3" in /usr/lib/python3/dist-packages (from scikit-build) (45.2.0)
Requirement already satisfied: distro in /usr/lib/python3/dist-packages (from scikit-build) (1.4.0)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.8/dist-packages/pyparsing-3.0.0b1-py3.8.egg (from packaging->scikit-build) (3.0.0b1)
Installing collected packages: scikit-build
Successfully installed scikit-build-0.11.1

$ python setup.py build

--------------------------------------------------------------------------------
-- Trying "Ninja" generator
--------------------------------
---------------------------
----------------------
-----------------
------------
-------
--
Not searching for unused variables given on the command line.
-- The C compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- The CXX compiler identification is GNU 9.3.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/data/archives/python-2.3.6/_cmake_test_compile/build
--
-------
------------
-----------------
----------------------
---------------------------
--------------------------------
-- Trying "Ninja" generator - success
--------------------------------------------------------------------------------

Configuring Project
  Working directory:
    /mnt/data/archives/python-2.3.6/_skbuild/linux-x86_64-3.8/cmake-build
  Command:
    cmake /mnt/data/archives/python-2.3.6 -G Ninja -DCMAKE_INSTALL_PREFIX:PATH=/mnt/data/archives/python-2.3.6/_skbuild/linux-x86_64-3.8/cmake-install -DPYTHON_EXECUTABLE:FILEPATH=/usr/bin/python -DPYTHON_VERSION_STRING:STRING=3.8.5 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPYTHON_LIBRARY:FILEPATH=/usr/lib/x86_64-linux-gnu/libpython3.8.so -DSKBUILD:BOOL=TRUE -DCMAKE_MODULE_PATH:PATH=/usr/local/lib/python3.8/dist-packages/skbuild/resources/cmake -DCMAKE_BUILD_TYPE:STRING=Release

-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python3: /usr/bin/python3.8 (found version "3.8.5") found components: Interpreter Development NumPy 
-- Found PythonInterp: /usr/bin/python (found version "3.8.5") 
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.8.so (found version "3.8.5") 
-- Found Cython: /usr/bin/cython3  
-- The project is built using scikit-build
-- Performing Test Weak Link MODULE -> SHARED (gnu_ld_ignore) - Success
_modinit_prefix:PyInit_
-- PROJECT_NAME: PDAL
-- PROJECT_NAME: PDAL
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/data/archives/python-2.3.6/_skbuild/linux-x86_64-3.8/cmake-build
[10/18] Building CXX object pdal/CMakeFiles/libpdalpython.dir/PyPipeline.cpp.o
In file included from /usr/include/python3.8/numpy/ndarraytypes.h:1830,
                 from /usr/include/python3.8/numpy/ndarrayobject.h:12,
                 from /usr/include/python3.8/numpy/arrayobject.h:4,
                 from ../../../pdal/PyPipeline.cpp:42:
/usr/include/python3.8/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
[11/18] Building CXX object pdal/CMakeFiles/libpdalpython.dir/libpdalpython.cxx.o
In file included from /usr/include/python3.8/numpy/ndarraytypes.h:1830,
                 from /usr/include/python3.8/numpy/ndarrayobject.h:12,
                 from /usr/include/python3.8/numpy/arrayobject.h:4,
                 from pdal/libpdalpython.cxx:619:
/usr/include/python3.8/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
[12/18] Building CXX object pdal/CMakeFiles/libpdalpython.dir/PyArray.cpp.o
In file included from /usr/include/python3.8/numpy/ndarraytypes.h:1830,
                 from ../../../pdal/PyArray.hpp:37,
                 from ../../../pdal/PyArray.cpp:35:
/usr/include/python3.8/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
[17/18] Install the project...
-- Install configuration: "Release"
-- Installing: /mnt/data/archives/python-2.3.6/_skbuild/linux-x86_64-3.8/cmake-install/pdal/libpdal_plugin_reader_numpy.so
-- Set runtime path of "/mnt/data/archives/python-2.3.6/_skbuild/linux-x86_64-3.8/cmake-install/pdal/libpdal_plugin_reader_numpy.so" to ""
-- Installing: /mnt/data/archives/python-2.3.6/_skbuild/linux-x86_64-3.8/cmake-install/pdal/libpdal_plugin_filter_python.so
-- Set runtime path of "/mnt/data/archives/python-2.3.6/_skbuild/linux-x86_64-3.8/cmake-install/pdal/libpdal_plugin_filter_python.so" to ""
-- Installing: /mnt/data/archives/python-2.3.6/_skbuild/linux-x86_64-3.8/cmake-install/pdal/libpdalpython.cpython-38-x86_64-linux-gnu.so
-- Set runtime path of "/mnt/data/archives/python-2.3.6/_skbuild/linux-x86_64-3.8/cmake-install/pdal/libpdalpython.cpython-38-x86_64-linux-gnu.so" to ""

copying pdal/array.py -> _skbuild/linux-x86_64-3.8/cmake-install/pdal/array.py
copying pdal/__init__.py -> _skbuild/linux-x86_64-3.8/cmake-install/pdal/__init__.py
copying pdal/dimension.py -> _skbuild/linux-x86_64-3.8/cmake-install/pdal/dimension.py
copying pdal/pio.py -> _skbuild/linux-x86_64-3.8/cmake-install/pdal/pio.py
copying pdal/pipeline.py -> _skbuild/linux-x86_64-3.8/cmake-install/pdal/pipeline.py

running build
running build_py
creating _skbuild/linux-x86_64-3.8/setuptools/lib
creating _skbuild/linux-x86_64-3.8/setuptools/lib/pdal
copying _skbuild/linux-x86_64-3.8/cmake-install/pdal/array.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/pdal
copying _skbuild/linux-x86_64-3.8/cmake-install/pdal/__init__.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/pdal
copying _skbuild/linux-x86_64-3.8/cmake-install/pdal/dimension.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/pdal
copying _skbuild/linux-x86_64-3.8/cmake-install/pdal/pio.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/pdal
copying _skbuild/linux-x86_64-3.8/cmake-install/pdal/pipeline.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/pdal
copying _skbuild/linux-x86_64-3.8/cmake-install/pdal/libpdal_plugin_reader_numpy.so -> _skbuild/linux-x86_64-3.8/setuptools/lib/pdal
copying _skbuild/linux-x86_64-3.8/cmake-install/pdal/libpdal_plugin_filter_python.so -> _skbuild/linux-x86_64-3.8/setuptools/lib/pdal
copying _skbuild/linux-x86_64-3.8/cmake-install/pdal/libpdalpython.cpython-38-x86_64-linux-gnu.so -> _skbuild/linux-x86_64-3.8/setuptools/lib/pdal
copied 5 files
running build_ext
hobu commented 3 years ago

very strange that it is not installing and finding the cython shim. Are you using a virtual environment or conda or something? I'm stumped.

gsakkis commented 3 years ago

No, just the system python on a Ubuntu 20.04 vm. Btw I uninstalled it and tried to reinstall it via pip again in case the problem was the (previously) missing scikit-build but still the same, it still didn't build libpdalpython.

gsakkis commented 3 years ago

I ran pip install with --verbose, the only thing that stands out is skipping skbuild (no CMakeLists.txt found). Related logs:

 Getting requirements to build wheel: started
  Running command /usr/bin/python3 /tmp/tmpf2yl8qc2 get_requires_for_build_wheel /tmp/tmp4o_6bilv
  skipping skbuild (no CMakeLists.txt found)
  running egg_info
  creating PDAL.egg-info
  writing PDAL.egg-info/PKG-INFO
  writing dependency_links to PDAL.egg-info/dependency_links.txt
  writing top-level names to PDAL.egg-info/top_level.txt
  writing manifest file 'PDAL.egg-info/SOURCES.txt'
  reading manifest file 'PDAL.egg-info/SOURCES.txt'
  writing manifest file 'PDAL.egg-info/SOURCES.txt'
  Getting requirements to build wheel: finished with status 'done'
    Created temporary directory: /tmp/pip-modern-metadata-mcmcoiw6
    Preparing wheel metadata: started
    Running command /usr/bin/python3 /tmp/tmpx84zw4pk prepare_metadata_for_build_wheel /tmp/tmp2smg593_
    skipping skbuild (no CMakeLists.txt found)
    running dist_info
    creating /tmp/pip-modern-metadata-mcmcoiw6/PDAL.egg-info
    writing /tmp/pip-modern-metadata-mcmcoiw6/PDAL.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-modern-metadata-mcmcoiw6/PDAL.egg-info/dependency_links.txt
    writing top-level names to /tmp/pip-modern-metadata-mcmcoiw6/PDAL.egg-info/top_level.txt
    writing manifest file '/tmp/pip-modern-metadata-mcmcoiw6/PDAL.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-modern-metadata-mcmcoiw6/PDAL.egg-info/SOURCES.txt'
    writing manifest file '/tmp/pip-modern-metadata-mcmcoiw6/PDAL.egg-info/SOURCES.txt'
    creating '/tmp/pip-modern-metadata-mcmcoiw6/PDAL.dist-info'
    Preparing wheel metadata: finished with status 'done'
  Source in /tmp/pip-install-dhw18kyh/pdal has version 2.3.6, which satisfies requirement pdal from https://files.pythonhosted.org/packages/1e/b3/1e736d42e0579048b1a4f9b686cd252a18d230603de2985220d9bc9cda4a/PDAL-2.3.6.tar.gz#sha256=b5e22c79043081e6b60d948af1f1036e4a5b1a503f843c0cb49cb669eff59c5e
  Removed pdal from https://files.pythonhosted.org/packages/1e/b3/1e736d42e0579048b1a4f9b686cd252a18d230603de2985220d9bc9cda4a/PDAL-2.3.6.tar.gz#sha256=b5e22c79043081e6b60d948af1f1036e4a5b1a503f843c0cb49cb669eff59c5e from build tracker '/tmp/pip-req-tracker-kp1yzcm1'
uilding wheels for collected packages: pdal
  Created temporary directory: /tmp/pip-wheel-8sve26bh
  Destination directory: /tmp/pip-wheel-8sve26bh
  Building wheel for pdal (PEP 517): started
  Running command /usr/bin/python3 /tmp/tmpkeayjl3_ build_wheel /tmp/tmp1yominh8
  skipping skbuild (no CMakeLists.txt found)
  running bdist_wheel
  running build
  running build_py
  creating _skbuild
  creating _skbuild/linux-x86_64-3.8
  creating _skbuild/linux-x86_64-3.8/setuptools
  creating _skbuild/linux-x86_64-3.8/setuptools/lib
  creating _skbuild/linux-x86_64-3.8/setuptools/lib/pdal
  copying pdal/__init__.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/pdal
  copying pdal/array.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/pdal
  copying pdal/pio.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/pdal
  copying pdal/dimension.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/pdal
  copying pdal/pipeline.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/pdal
  copied 5 files
  installing to _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/wheel
  running install
  running install_lib
  creating _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64
  creating _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/wheel
  creating _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/wheel/pdal
  copying _skbuild/linux-x86_64-3.8/setuptools/lib/pdal/__init__.py -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/wheel/pdal
  copying _skbuild/linux-x86_64-3.8/setuptools/lib/pdal/array.py -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/wheel/pdal
  copying _skbuild/linux-x86_64-3.8/setuptools/lib/pdal/pio.py -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/wheel/pdal
  copying _skbuild/linux-x86_64-3.8/setuptools/lib/pdal/dimension.py -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/wheel/pdal
  copying _skbuild/linux-x86_64-3.8/setuptools/lib/pdal/pipeline.py -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/wheel/pdal
  copied 5 files
 running install_egg_info
  running egg_info
  writing PDAL.egg-info/PKG-INFO
  writing dependency_links to PDAL.egg-info/dependency_links.txt
  writing top-level names to PDAL.egg-info/top_level.txt
  reading manifest file 'PDAL.egg-info/SOURCES.txt'
  writing manifest file 'PDAL.egg-info/SOURCES.txt'
  Copying PDAL.egg-info to _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/wheel/PDAL-2.3.6-py3.8.egg-info
  running install_scripts
  copied 0 files
  creating _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/wheel/PDAL-2.3.6.dist-info/WHEEL
  creating '/tmp/pip-wheel-8sve26bh/tmpgm2gmi6q/PDAL-2.3.6-py3-none-any.whl' and adding '_skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/wheel' to it
  adding 'pdal/__init__.py'
  adding 'pdal/array.py'
  adding 'pdal/dimension.py'
  adding 'pdal/pio.py'
  adding 'pdal/pipeline.py'
  adding 'PDAL-2.3.6.dist-info/METADATA'
  adding 'PDAL-2.3.6.dist-info/WHEEL'
  adding 'PDAL-2.3.6.dist-info/top_level.txt'
  adding 'PDAL-2.3.6.dist-info/RECORD'
  removing _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/wheel
  Building wheel for pdal (PEP 517): finished with status 'done'
  Created wheel for pdal: filename=PDAL-2.3.6-py3-none-any.whl size=6467 sha256=d9f9498e669ae2216edeb180246e1493d77b5a9a97d5c8bb1b7610621696e7db
  Stored in directory: /tmp/pip-ephem-wheel-cache-_qpcxnxv/wheels/12/ef/4f/30f6a1126d2a413b49a6208034bf1262b219b848ebd3025ebd
Successfully built pdal
Installing collected packages: pdal
  Created temporary directory: /tmp/pip-unpacked-wheel-3ks24sj1

Successfully installed pdal-2.3.6
Cleaning up...
  Removing source in /tmp/pip-install-dhw18kyh/pdal
Removed build tracker: '/tmp/pip-req-tracker-kp1yzcm1'
hobu commented 3 years ago

Strange. If _skbuild isn't being built, the cython shim isn't being compiled. Please compare to our CI setup and look if you see any interesting discrepancies https://github.com/PDAL/python/blob/master/.github/workflows/build.yml

fernandojsg commented 3 years ago

@gsakkis I'm still getting the same issue you described after trying to install pdal using pip. Is this expected to be corrected on the next release with the PR you just merged? Or should I follow any other steps in order to get it running? Any help will be more than welcome :) Thanks!

hobu commented 3 years ago

I'll make a new release today.

fernandojsg commented 3 years ago

superb! thanks! :)

hobu commented 3 years ago

https://pypi.org/project/PDAL/ done.

fernandojsg commented 3 years ago

@hobu I just tested it and it worked! thanks a lot :)