search

apache / arrow

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
https://arrow.apache.org/
Apache License 2.0
14.42k stars 3.51k forks source link

[Python] Fails to build PyArrow@12.0.0 on RH8 #35586

Open orliac opened 1 year ago

orliac commented 1 year ago

Describe the bug, including details regarding any error messages, version, and platform.

When installing arrow@12.0.0 from source using GCC 11.3.0 on a RH8 machine (pip 22.3.1, Python 3.9, using Spack) I bump into the following issue:

  -- Running cmake --build for PyArrow
  cmake --build . --config release --
  [  1%] Compiling Cython CXX source for lib...

  Error compiling Cython file:
  ------------------------------------------------------------
  ...

  cdef extern from "Python.h":
      int PySlice_Check(object)

  cdef int check_status(const CStatus& status) except -1 nogil
                                                        ^
  ------------------------------------------------------------

  pyarrow/lib.pxd:67:55: Syntax error in C variable declaration

Can someone see what the issue could be? Thank you.

Below the full log.

  -- Running cmake for PyArrow
  cmake -DCMAKE_INSTALL_PREFIX=/tmp/orliac/spack-stage/spack-stage-py-pyarrow-12.0.0-ghhdfuencjj7ztv4eywxmqn6emhq56u7/spack-src/build/lib.linux-x86_64-3.9/pyarrow -DPYTHON_EXECUTABLE=/work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/python-3.9.15-zsmss6eznpwn5qtkdkxpli3qhsr54fld/bin/python3.9 -DPython3_EXECUTABLE=/work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/python-3.9.15-zsmss6eznpwn5qtkdkxpli3qhsr54fld/bin/python3.9 -DPYARROW_CXXFLAGS= -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_SUBSTRAIT=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ACERO=on -DPYARROW_BUILD_DATASET=on -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PARQUET_ENCRYPTION=off -DPYARROW_BUILD_GCS=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_CYTHON_CPP=off -DPYARROW_GENERATE_COVERAGE=off -DCMAKE_BUILD_TYPE=release /tmp/orliac/spack-stage/spack-stage-py-pyarrow-12.0.0-ghhdfuencjj7ztv4eywxmqn6emhq56u7/spack-src
  -- The C compiler identification is GNU 11.3.0
  -- The CXX compiler identification is GNU 11.3.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /work/backup/ska/orliac/spack/lib/spack/env/gcc/gcc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /work/backup/ska/orliac/spack/lib/spack/env/gcc/g++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Using ccache: /usr/bin/ccache
  -- System processor: x86_64
  -- Performing Test CXX_SUPPORTS_SSE4_2
  -- Performing Test CXX_SUPPORTS_SSE4_2 - Success
  -- Performing Test CXX_SUPPORTS_AVX2
  -- Performing Test CXX_SUPPORTS_AVX2 - Success
  -- Performing Test CXX_SUPPORTS_AVX512
  -- Performing Test CXX_SUPPORTS_AVX512 - Success
  -- Arrow build warning level: PRODUCTION
  -- Using ld linker
  -- Build Type: RELEASE
  -- CMAKE_C_FLAGS:  -Wall -fno-semantic-interposition -msse4.2  -fdiagnostics-color=always  -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized
  -- CMAKE_CXX_FLAGS:  -Wno-noexcept-type  -Wall -fno-semantic-interposition -msse4.2  -fdiagnostics-color=always  -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized
  -- Generator: Unix Makefiles
  -- Build output directory: /tmp/orliac/spack-stage/spack-stage-py-pyarrow-12.0.0-ghhdfuencjj7ztv4eywxmqn6emhq56u7/spack-src/build/temp.linux-x86_64-3.9/release
  -- Found Python3: /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/python-3.9.15-zsmss6eznpwn5qtkdkxpli3qhsr54fld/bin/python3.9 (found version "3.9.15") found components: Interpreter Development.Module NumPy
  -- Found Python3Alt: /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/python-3.9.15-zsmss6eznpwn5qtkdkxpli3qhsr54fld/bin/python3.9
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
  -- Check if compiler accepts -pthread
  -- Check if compiler accepts -pthread - yes
  -- Found Threads: TRUE
  CMake Warning at /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/arrow-12.0.0-b6lftge64at2hocynsnwm2u5m3u6bcce/lib64/cmake/Arrow/FindThriftAlt.cmake:56 (find_package):
    By not providing "FindThrift.cmake" in CMAKE_MODULE_PATH this project has
    asked CMake to find a package configuration file provided by "Thrift", but
    CMake did not find one.

    Could not find a package configuration file provided by "Thrift" with any
    of the following names:

      ThriftConfig.cmake
      thrift-config.cmake

    Add the installation prefix of "Thrift" to CMAKE_PREFIX_PATH or set
    "Thrift_DIR" to a directory containing one of the above files.  If "Thrift"
    provides a separate development package or SDK, be sure it has been
    installed.
  Call Stack (most recent call first):
    /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/cmake-3.24.3-5pf3twpa3mag2ln3cgxu3mkjbarasb4a/share/cmake-3.24/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
    /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/arrow-12.0.0-b6lftge64at2hocynsnwm2u5m3u6bcce/lib64/cmake/Arrow/ArrowConfig.cmake:105 (find_dependency)
    CMakeLists.txt:261 (find_package)

  -- Checking for module 'thrift'
  --   Found thrift, version 0.16.0
  -- Found ThriftAlt: /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/thrift-0.16.0-aahpf6q4yv4lvcypwegxj3xh2bjsehb4/lib/libthrift.so (found version "0.16.0")
  -- Found utf8proc: /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/utf8proc-2.8.0-36yitvl7cdvcnzgepfzbaddy2pr4dqaq/lib64/libutf8proc.so (found version "2.8.0")
  -- Arrow version: 12.0.0
  -- Found the Arrow shared library: /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/arrow-12.0.0-b6lftge64at2hocynsnwm2u5m3u6bcce/lib64/libarrow.so.1200.0.0
  -- Found the Arrow import library: ARROW_IMPORT_LIB-NOTFOUND
  -- Found the Arrow static library: /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/arrow-12.0.0-b6lftge64at2hocynsnwm2u5m3u6bcce/lib64/libarrow.a
  -- Parquet version: 12.0.0
  -- Found the Parquet shared library: /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/arrow-12.0.0-b6lftge64at2hocynsnwm2u5m3u6bcce/lib64/libparquet.so.1200.0.0
  -- Found the Parquet import library: PARQUET_IMPORT_LIB-NOTFOUND
  -- Found the Parquet static library: /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/arrow-12.0.0-b6lftge64at2hocynsnwm2u5m3u6bcce/lib64/libparquet.a
  -- ArrowAcero version: 12.0.0
  -- Found the ArrowAcero shared library: /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/arrow-12.0.0-b6lftge64at2hocynsnwm2u5m3u6bcce/lib64/libarrow_acero.so.1200.0.0
  -- Found the ArrowAcero import library: ARROW_ACERO_IMPORT_LIB-NOTFOUND
  -- Found the ArrowAcero static library: /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/arrow-12.0.0-b6lftge64at2hocynsnwm2u5m3u6bcce/lib64/libarrow_acero.a
  -- ArrowDataset version: 12.0.0
  -- Found the ArrowDataset shared library: /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/arrow-12.0.0-b6lftge64at2hocynsnwm2u5m3u6bcce/lib64/libarrow_dataset.so.1200.0.0
  -- Found the ArrowDataset import library: ARROW_DATASET_IMPORT_LIB-NOTFOUND
  -- Found the ArrowDataset static library: /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/arrow-12.0.0-b6lftge64at2hocynsnwm2u5m3u6bcce/lib64/libarrow_dataset.a
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /tmp/orliac/spack-stage/spack-stage-py-pyarrow-12.0.0-ghhdfuencjj7ztv4eywxmqn6emhq56u7/spack-src/build/temp.linux-x86_64-3.9
  -- Finished cmake for PyArrow
  -- Running cmake --build for PyArrow
  cmake --build . --config release --
  [  1%] Compiling Cython CXX source for lib...

  Error compiling Cython file:
  ------------------------------------------------------------
  ...

  cdef extern from "Python.h":
      int PySlice_Check(object)

  cdef int check_status(const CStatus& status) except -1 nogil
                                                        ^
  ------------------------------------------------------------

  pyarrow/lib.pxd:67:55: Syntax error in C variable declaration

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  ArrowIOError = IOError

  # This function could be written directly in C++ if we didn't
  # define Arrow-specific subclasses (ArrowInvalid etc.)
  cdef int check_status(const CStatus& status) except -1 nogil:
                                                        ^
  ------------------------------------------------------------

  pyarrow/error.pxi:82:55: Syntax error in C variable declaration
  gmake[2]: *** [CMakeFiles/lib_pyx.dir/build.make:71: CMakeFiles/lib_pyx] Error 1
  gmake[1]: *** [CMakeFiles/Makefile2:146: CMakeFiles/lib_pyx.dir/all] Error 2
  gmake: *** [Makefile:136: all] Error 2
  error: command '/work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/cmake-3.24.3-5pf3twpa3mag2ln3cgxu3mkjbarasb4a/bin/cmake' failed with exit code 2
  error: subprocess-exited-with-error

  × Building wheel for pyarrow (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/python-3.9.15-zsmss6eznpwn5qtkdkxpli3qhsr54fld/bin/python3.9 /work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/py-pip-22.3.1-oiljmniioyay6w4x6nlhbge3uqexfzyu/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp4vqpbc1n
  cwd: /tmp/orliac/spack-stage/spack-stage-py-pyarrow-12.0.0-ghhdfuencjj7ztv4eywxmqn6emhq56u7/spack-src
  Building wheel for pyarrow (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects
Exception information:
Traceback (most recent call last):
  File "/work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/py-pip-22.3.1-oiljmniioyay6w4x6nlhbge3uqexfzyu/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
  File "/work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/py-pip-22.3.1-oiljmniioyay6w4x6nlhbge3uqexfzyu/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/work/backup/ska/orliac/spack/opt/spack/linux-rhel8-icelake/gcc-11.3.0/py-pip-22.3.1-oiljmniioyay6w4x6nlhbge3uqexfzyu/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 464, in run
    raise InstallationError(
pip._internal.exceptions.InstallationError: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects
Removed build tracker: '/tmp/pip-build-tracker-il8ou_m6'

Component(s)

C++, Python

westonpace commented 1 year ago

Hmm, do you know what version of cython you have installed? To support cython 3 we moved from:

cdef int check_status(const CStatus& status) nogil except -1:

to...

cdef int check_status(const CStatus& status) except -1 nogil:

Both styles were fine with cython version 0.29.30. I wonder if some older versions of cython don't recognize the "nogil at the end" syntax.

orliac commented 1 year ago

Thanks @westonpace Indeed, I could solve the issue using a more recent version of Cython. With version 0.29.33 it compiles smoothly.

jorisvandenbossche commented 1 year ago

@orliac do you know which version of cython you used before? (when it was failing)

Because we currently have "cython >= 0.29.31" specified in our build dependencies, and so wondering if we need to bump that version there