Trouble installing from PyPI: `error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘fields’`

[tayden]

I'm having a time trying to build the PDAL library from source, and then install python-pdal with pip in a Docker container. This was working last week, so I strongly suspect my issues are related to the release of Numpy 2.0.0 and subsequent deprecation of some of their C APIs.

Strangely, I still can't get it to work when pinning Numpy to some version <2 (that still meets the python-pdal dependency requirements).

The whole error printout is:

------
 > [6/8] RUN pip install pdal:
5.518 Collecting pdal
5.778   Downloading pdal-3.4.3.tar.gz (89 kB)
5.858      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.5/89.5 kB 1.0 MB/s eta 0:00:00
5.863   Installing build dependencies: started
14.25   Installing build dependencies: finished with status 'done'
14.25   Getting requirements to build wheel: started
14.31   Getting requirements to build wheel: finished with status 'done'
14.31   Preparing metadata (pyproject.toml): started
14.38   Preparing metadata (pyproject.toml): finished with status 'done'
14.38 Requirement already satisfied: numpy>=1.22 in /usr/local/lib/python3.11/site-packages (from pdal) (1.22.0)
14.38 Building wheels for collected packages: pdal
14.38   Building wheel for pdal (pyproject.toml): started
17.74   Building wheel for pdal (pyproject.toml): finished with status 'error'
17.74   error: subprocess-exited-with-error
17.74
17.74   × Building wheel for pdal (pyproject.toml) did not run successfully.
17.74   │ exit code: 1
17.74   ╰─> [42 lines of output]
17.74       *** scikit-build-core 0.9.6 using CMake 3.25.1 (wheel)
17.74       *** Configuring CMake...
17.74       loading initial cache file build/cp311-cp311-linux_x86_64/CMakeInit.txt
17.74       -- The C compiler identification is GNU 12.2.0
17.74       -- The CXX compiler identification is GNU 12.2.0
17.74       -- Detecting C compiler ABI info
17.74       -- Detecting C compiler ABI info - done
17.74       -- Check for working C compiler: /usr/bin/cc - skipped
17.74       -- Detecting C compile features
17.74       -- Detecting C compile features - done
17.74       -- Detecting CXX compiler ABI info
17.74       -- Detecting CXX compiler ABI info - done
17.74       -- Check for working CXX compiler: /usr/bin/c++ - skipped
17.74       -- Detecting CXX compile features
17.74       -- Detecting CXX compile features - done
17.74       -- Found Python3: /usr/local/bin/python (found version "3.11.9") found components: Interpreter Development.Module NumPy
17.74       -- Performing Test HAS_FLTO
17.74       -- Performing Test HAS_FLTO - Success
17.74       -- Found pybind11: /tmp/pip-build-env-m9teg40_/overlay/lib/python3.11/site-packages/pybind11/include (found version "2.12.0")
17.74       -- Configuring done
17.74       -- Generating done
17.74       -- Build files have been written to: /tmp/pip-install-_l8qrq6n/pdal_a380f1311cc54674ac4cc5d4e56ba631/build/cp311-cp311-linux_x86_64
17.74       *** Building project with Ninja...
17.74       [1/5] Building CXX object CMakeFiles/libpdalpython.dir/src/pdal/PyArray.cpp.o
17.74       FAILED: CMakeFiles/libpdalpython.dir/src/pdal/PyArray.cpp.o
17.74       /usr/bin/c++ -Dlibpdalpython_EXPORTS -I/tmp/pip-build-env-m9teg40_/overlay/lib/python3.11/site-packages/numpy/_core/include -isystem /usr/local/include/python3.11 -isystem /tmp/pip-build-env-m9teg40_/overlay/lib/python3.11/site-packages/pybind11/include -O3 -DNDEBUG -fPIC -fvisibility=hidden -flto -fno-fat-lto-objects -std=c++17 -MD -MT CMakeFiles/libpdalpython.dir/src/pdal/PyArray.cpp.o -MF CMakeFiles/libpdalpython.dir/src/pdal/PyArray.cpp.o.d -o CMakeFiles/libpdalpython.dir/src/pdal/PyArray.cpp.o -c /tmp/pip-install-_l8qrq6n/pdal_a380f1311cc54674ac4cc5d4e56ba631/src/pdal/PyArray.cpp
17.74       /tmp/pip-install-_l8qrq6n/pdal_a380f1311cc54674ac4cc5d4e56ba631/src/pdal/PyArray.cpp: In constructor ‘pdal::python::Array::Array(PyArrayObject*)’:
17.74       /tmp/pip-install-_l8qrq6n/pdal_a380f1311cc54674ac4cc5d4e56ba631/src/pdal/PyArray.cpp:100:29: error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘fields’
17.74         100 |     int numFields = (dtype->fields == Py_None) ?
17.74             |                             ^~~~~~
17.74       /tmp/pip-install-_l8qrq6n/pdal_a380f1311cc54674ac4cc5d4e56ba631/src/pdal/PyArray.cpp:102:45: error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘fields’
17.74         102 |         static_cast<int>(PyDict_Size(dtype->fields));
17.74             |                                             ^~~~~~
17.74       /tmp/pip-install-_l8qrq6n/pdal_a380f1311cc54674ac4cc5d4e56ba631/src/pdal/PyArray.cpp:113:39: error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘fields’
17.74         113 |         PyObject *names_dict = dtype->fields;
17.74             |                                       ^~~~~~
17.74       [2/5] Building CXX object CMakeFiles/libpdalpython.dir/src/pdal/StreamableExecutor.cpp.o
17.74       [3/5] Building CXX object CMakeFiles/libpdalpython.dir/src/pdal/PyPipeline.cpp.o
17.74       [4/5] Building CXX object CMakeFiles/libpdalpython.dir/src/pdal/libpdalpython.cpp.o
17.74       ninja: build stopped: subcommand failed.
17.74
17.74       *** CMake build failed
17.74       [end of output]
17.74
17.74   note: This error originates from a subprocess, and is likely not a problem with pip.
17.74   ERROR: Failed building wheel for pdal
17.74 Failed to build pdal
17.74 ERROR: Could not build wheels for pdal, which is required to install pyproject.toml-based projects
------
Dockerfile:32
--------------------
  30 |
  31 |     # Install Python pdal package
  32 | >>> RUN pip install pdal
  33 |
  34 |     # Clean up unnecessary files to reduce image size
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install pdal" did not complete successfully: exit code: 1

The Dockerfile being built looks like this (simplified from the original)

  # Stage 1: Build PDAL
  FROM python:3.11

  # Install build dependencies
  RUN apt-get update && apt-get install -y \
      cmake \
      build-essential \
      ninja-build \
      wget \
      libgdal-dev \
      && apt-get clean

  # I was sure that pinning Numpy would make it work, but no luck
  RUN pip install "numpy==1.22"

  # Download and extract PDAL source code
  RUN wget https://github.com/PDAL/PDAL/releases/download/2.7.1/PDAL-2.7.1-src.tar.bz2 -O /tmp/PDAL-2.7.1-src.tar.bz2 \
      && mkdir -p /tmp/pdal-src \
      && tar -xjf /tmp/PDAL-2.7.1-src.tar.bz2 -C /tmp/pdal-src --strip-components=1 \
      && rm /tmp/PDAL-2.7.1-src.tar.bz2

  # Build and install PDAL using Ninja
  RUN cd /tmp/pdal-src \
      && cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local . \
      && ninja \
      && ninja install

  # Set environment variables for runtime
  ENV LD_LIBRARY_PATH=/usr/local/lib:${LD_LIBRARY_PATH}
  ENV CMAKE_PREFIX_PATH=/usr/local

  # Install Python pdal package
  RUN pip install pdal

  # Clean up unnecessary files to reduce image size
  RUN apt-get remove --purge -y \
      wget \
      build-essential \
      cmake \
      git \
      && apt-get autoremove -y && apt-get clean

  # Verify installation
  RUN ldconfig && python -c "import pdal; print('PDAL imported successfully')"

Thanks in advance for any help!

[hobu]

I'll take a look. I had made some changes for Numpy 2.0 in #167, but maybe there is more to do.

[iseong83]

@hobu I am having the same issue with pdal==3.1.2. Is it possible to pin the Numpy version for the old pdal releases?

[hobu]

I'm confused as to what you need. Do you need a version less than 1.22? https://github.com/PDAL/python/blob/main/pyproject.toml#L29 ?

[tayden]

Any numpy version is fine for me. Installing numpy==1.22 was an attempt to get pip install pdal to run, but didn't fix the error.

[iseong83]

For my side, I believe I need numpy<2. When I install pdal using pip install pdal, it seems I cannot configure which numpy should install. Pip keep installing numpy==2.0.0

[hobu]

Please try https://github.com/PDAL/python/pull/170

[tayden]

Works great! Thanks @hobu!

[hobu]

ok, I'll bump and push releases today. sorry for the trouble.

[tayden]

No worries! Thanks for the quick fix!

[tayden]

For my side, I believe I need numpy<2. When I install pdal using pip install pdal, it seems I cannot configure which numpy should install. Pip keep installing numpy==2.0.0

@iseong83 you should be able to specify your numpy version by installing it separately from pdal (instead of letting pdal choose the version) i.e. try pip install "numpy<2" pdal