Fail to build pyarrow on Windows 10 from source

[oraix]

I'm following the instruction in build pyarrow on windows, but failed at the step of "python setup.py build_ext --inplace".

C:\dev\tmp\arrow\python>python setup.py build_ext --inplace C:\softs\dev\anaconda3\envs\rq\lib\site-packages\setuptools_scm\git.py:105: UserWarning: "C:\dev\tmp\arrow" is shallow and may cause errors warnings.warn(f'"{wd.path}" is shallow and may cause errors') running build_ext creating C:\dev\tmp\arrow\python\build creating C:\dev\tmp\arrow\python\build\temp.win-amd64-cpython-38 -- Running cmake for pyarrow cmake -DPYTHON_EXECUTABLE=C:\softs\dev\anaconda3\envs\rq\python.exe -DPython3_EXECUTABLE=C:\softs\dev\anaconda3\envs\rq\python.exe "" -G "Visual Studio 15 2017 Win64" -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PARQUET_ENCRYPTION=off -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release C:\dev\tmp\arrow\python CMake Warning: Ignoring empty string ("") provided on the command line.

-- Selecting Windows SDK version 10.0.17763.0 to target Windows 10.0.19044. -- The C compiler identification is MSVC 19.16.27045.0 -- The CXX compiler identification is MSVC 19.16.27045.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC/Tools/MSVC/14.16.27023/bin/Hostx86/x64/cl.exe - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC/Tools/MSVC/14.16.27023/bin/Hostx86/x64/cl.exe - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- System processor: AMD64 -- Performing Test CXX_SUPPORTS_AVX2 -- Performing Test CXX_SUPPORTS_AVX2 - Success -- Performing Test CXX_SUPPORTS_AVX512 -- Performing Test CXX_SUPPORTS_AVX512 - Success -- Arrow build warning level: PRODUCTION Configured for RELEASE build (set with cmake -DCMAKE_BUILD_TYPE={release,debug,...}) -- Build Type: RELEASE -- Generator: Visual Studio 15 2017 Win64 -- Build output directory: C:/dev/tmp/arrow/python/build/temp.win-amd64-cpython-38/ -- Found Python3: C:\softs\dev\anaconda3\envs\rq\python.exe (found version "3.8.13") found components: Interpreter Development.Module NumPy -- Found Python3Alt: C:\softs\dev\anaconda3\envs\rq\python.exe -- Found PkgConfig: C:/softs/dev/anaconda3/envs/rq/Library/bin/pkg-config.exe (found version "0.29.2") _CMake Error at C:/softs/dev/anaconda3/envs/rq/Library/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:230 (message): Could NOT find Arrow (missing: ARROW_INCLUDE_DIR ARROW_FULL_SO_VERSION ARROW_SOVERSION) Call Stack (most recent call first): C:/softs/dev/anaconda3/envs/rq/Library/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) C:/dev/tmp/arrow/cpp/cmake_modules/FindArrow.cmake:450 (find_package_handle_standard_args) C:/dev/tmp/arrow/cpp/cmake_modules/FindArrowPython.cmake:46 (find_package) CMakeLists.txt:231 (find_package)

-- Configuring incomplete, errors occurred! See also "C:/dev/tmp/arrow/python/build/temp.win-amd64-cpython-38/CMakeFiles/CMakeOutput.log". error: command 'C:\softs\dev\anaconda3\envs\rq\Library\bin\cmake.exe' failed with exit code 1

Could anybody help me out? Thanks for your kindly help!

PS. I try to build the package of pyarrow, because when I using the pyarrow 8.0.* installed by pip, I'll get the error:

Error compiling Cython file:
------------------------------------------------------------
...

cdef class OrcFileFormat(FileFormat):

    def __init__(self):
        self.init(shared_ptr[CFileFormat](new COrcFileFormat()))
                                             ^
------------------------------------------------------------

C:\softs\dev\anaconda3\envs\rqalpha\lib\site-packages\pyarrow\_dataset_orc.pyx:32:46: Operation only allowed in c++
_dataset_orc.c
C:\Users\avatar\.pyxbld\temp.win-amd64-cpython-38\Release\pyrex\pyarrow\_dataset_orc.c(1): fatal error C1189: #error:  Do not use this file, it is the result of a failed Cython compilation.

[AlenkaF]

Thanks for reporting @oraix. Have you made any progress? If not, can you also share the steps and output of the Arrow C++ build?

[pitrou]

@oraix Some binary packages are available for Windows, so it's unexpected that pip is trying to compile from source. Can you post the output of pip install --prefer-binary pyarrow?