Closed tcrasset closed 7 months ago
Could you also show make install
log?
/home/tom/Documents/dev/arrow/cpp/build/src/arrow/ArrowConfig.cmake:99 (find_dependency)
/home/tom/Documents/dev/arrow/cpp/build/src/arrow/ArrowConfig.cmake:122 (arrow_find_dependencies)
This is strange. We should use /home/tom/Documents/dev/dist/lib/cmake/Arrow/ArrowConfig.cmake
instead.
Thank you for your quick response. I added it to the main comment.
Could you try the following patterns?
1:
$ python setup.py clean
$ CMAKE_PREFIX_PATH=$ARROW_HOME python setup.py build_ext --inplace
2:
$ python setup.py clean
$ CMAKE_PREFIX_PATH=/home/tom/Documents/dev/dist python setup.py build_ext --inplace
3:
$ python setup.py clean
$ PYARROW_CMAKE_OPTIONS="-DCMAKE_PREFIX_PATH=$ARROW_HOME" python setup.py build_ext --inplace
4:
$ python setup.py clean
$ PYARROW_CMAKE_OPTIONS="-DCMAKE_PREFIX_PATH=/home/tom/Documents/dev/dist" python setup.py build_ext --inplace
Same end result, though I see that 3. and 4. did change the options for cmake, while the first two did not:
arrow/python on main [!?] via △ v3.27.9 via 🐍 v3.8.10 on ☁️ octopize (eu-west-3) on ☁️ tom@octopize.io
❯ python setup.py clean
running clean
removing 'build/temp.linux-x86_64-cpython-38' (and everything under it)
arrow/python on main [!?] via △ v3.27.9 via 🐍 v3.8.10 on ☁️ octopize (eu-west-3) on ☁️ tom@octopize.io
❯ echo $ARROW_HOME
arrow/python on main [!?] via △ v3.27.9 via 🐍 v3.8.10 on ☁️ octopize (eu-west-3) on ☁️ tom@octopize.io
❯ export ARROW_HOME=/home/tom/Documents/dev/dist
arrow/python on main [!?] via △ v3.27.9 via 🐍 v3.8.10 on ☁️ octopize (eu-west-3) on ☁️ tom@octopize.io
❯ CMAKE_PREFIX_PATH=$ARROW_HOME python setup.py build_ext --inplace
running build_ext
creating /home/tom/Documents/dev/arrow/python/build/temp.linux-x86_64-cpython-38
-- Running cmake for PyArrow
cmake -DCMAKE_INSTALL_PREFIX=/home/tom/Documents/dev/arrow/python/pyarrow -DPYTHON_EXECUTABLE=/usr/bin/python -DPython3_EXECUTABLE=/usr/bin/python -DPYARROW_CXXFLAGS= -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_SUBSTRAIT=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ACERO=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PARQUET_ENCRYPTION=off -DPYARROW_BUILD_GCS=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_CYTHON_CPP=off -DPYARROW_GENERATE_COVERAGE=off -DCMAKE_BUILD_TYPE=release /home/tom/Documents/dev/arrow/python
-- System processor: x86_64
-- Arrow build warning level: PRODUCTION
-- Using ld linker
-- Build Type: RELEASE
-- CMAKE_C_FLAGS: -Wall -fno-semantic-interposition -msse4.2 -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized
-- CMAKE_CXX_FLAGS: -Wno-noexcept-type -Wall -fno-semantic-interposition -msse4.2 -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized
-- Generator: Unix Makefiles
-- Build output directory: /home/tom/Documents/dev/arrow/python/build/release
CMake Error at /home/tom/.local/lib/python3.8/site-packages/cmake/data/share/cmake-3.27/Modules/CMakeFindDependencyMacro.cmake:76 (find_package):
By not providing "FindOpenSSLAlt.cmake" in CMAKE_MODULE_PATH this project
has asked CMake to find a package configuration file provided by
"OpenSSLAlt", but CMake did not find one.
Could not find a package configuration file provided by "OpenSSLAlt" with
any of the following names:
OpenSSLAltConfig.cmake
opensslalt-config.cmake
Add the installation prefix of "OpenSSLAlt" to CMAKE_PREFIX_PATH or set
"OpenSSLAlt_DIR" to a directory containing one of the above files. If
"OpenSSLAlt" provides a separate development package or SDK, be sure it has
been installed.
Call Stack (most recent call first):
/home/tom/Documents/dev/arrow/cpp/build/src/arrow/ArrowConfig.cmake:99 (find_dependency)
/home/tom/Documents/dev/arrow/cpp/build/src/arrow/ArrowConfig.cmake:122 (arrow_find_dependencies)
CMakeLists.txt:268 (find_package)
-- Configuring incomplete, errors occurred!
error: command '/home/tom/.local/bin/cmake' failed with exit code 1
arrow/python on main [!?] via △ v3.27.9 via 🐍 v3.8.10 on ☁️ octopize (eu-west-3) on ☁️ tom@octopize.io
❯ python setup.py clean
running clean
removing 'build/temp.linux-x86_64-cpython-38' (and everything under it)
arrow/python on main [!?] via △ v3.27.9 via 🐍 v3.8.10 on ☁️ octopize (eu-west-3) on ☁️ tom@octopize.io
❯ CMAKE_PREFIX_PATH=/home/tom/Documents/dev/dist python setup.py build_ext --inplace
running build_ext
creating /home/tom/Documents/dev/arrow/python/build/temp.linux-x86_64-cpython-38
-- Running cmake for PyArrow
cmake -DCMAKE_INSTALL_PREFIX=/home/tom/Documents/dev/arrow/python/pyarrow -DPYTHON_EXECUTABLE=/usr/bin/python -DPython3_EXECUTABLE=/usr/bin/python -DPYARROW_CXXFLAGS= -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_SUBSTRAIT=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ACERO=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PARQUET_ENCRYPTION=off -DPYARROW_BUILD_GCS=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_CYTHON_CPP=off -DPYARROW_GENERATE_COVERAGE=off -DCMAKE_BUILD_TYPE=release /home/tom/Documents/dev/arrow/python
-- System processor: x86_64
-- Arrow build warning level: PRODUCTION
-- Using ld linker
-- Build Type: RELEASE
-- CMAKE_C_FLAGS: -Wall -fno-semantic-interposition -msse4.2 -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized
-- CMAKE_CXX_FLAGS: -Wno-noexcept-type -Wall -fno-semantic-interposition -msse4.2 -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized
-- Generator: Unix Makefiles
-- Build output directory: /home/tom/Documents/dev/arrow/python/build/release
CMake Error at /home/tom/.local/lib/python3.8/site-packages/cmake/data/share/cmake-3.27/Modules/CMakeFindDependencyMacro.cmake:76 (find_package):
By not providing "FindOpenSSLAlt.cmake" in CMAKE_MODULE_PATH this project
has asked CMake to find a package configuration file provided by
"OpenSSLAlt", but CMake did not find one.
Could not find a package configuration file provided by "OpenSSLAlt" with
any of the following names:
OpenSSLAltConfig.cmake
opensslalt-config.cmake
Add the installation prefix of "OpenSSLAlt" to CMAKE_PREFIX_PATH or set
"OpenSSLAlt_DIR" to a directory containing one of the above files. If
"OpenSSLAlt" provides a separate development package or SDK, be sure it has
been installed.
Call Stack (most recent call first):
/home/tom/Documents/dev/arrow/cpp/build/src/arrow/ArrowConfig.cmake:99 (find_dependency)
/home/tom/Documents/dev/arrow/cpp/build/src/arrow/ArrowConfig.cmake:122 (arrow_find_dependencies)
CMakeLists.txt:268 (find_package)
-- Configuring incomplete, errors occurred!
error: command '/home/tom/.local/bin/cmake' failed with exit code 1
arrow/python on main [!?] via △ v3.27.9 via 🐍 v3.8.10 on ☁️ octopize (eu-west-3) on ☁️ tom@octopize.io
❯ python setup.py clean
running clean
removing 'build/temp.linux-x86_64-cpython-38' (and everything under it)
arrow/python on main [!?] via △ v3.27.9 via 🐍 v3.8.10 on ☁️ octopize (eu-west-3) on ☁️ tom@octopize.io
❯ PYARROW_CMAKE_OPTIONS="-DCMAKE_PREFIX_PATH=$ARROW_HOME" python setup.py build_ext --inplace
running build_ext
creating /home/tom/Documents/dev/arrow/python/build/temp.linux-x86_64-cpython-38
-- Running cmake for PyArrow
cmake -DCMAKE_PREFIX_PATH=/home/tom/Documents/dev/dist -DCMAKE_INSTALL_PREFIX=/home/tom/Documents/dev/arrow/python/pyarrow -DPYTHON_EXECUTABLE=/usr/bin/python -DPython3_EXECUTABLE=/usr/bin/python -DPYARROW_CXXFLAGS= -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_SUBSTRAIT=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ACERO=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PARQUET_ENCRYPTION=off -DPYARROW_BUILD_GCS=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_CYTHON_CPP=off -DPYARROW_GENERATE_COVERAGE=off -DCMAKE_BUILD_TYPE=release /home/tom/Documents/dev/arrow/python
-- System processor: x86_64
-- Arrow build warning level: PRODUCTION
-- Using ld linker
-- Build Type: RELEASE
-- CMAKE_C_FLAGS: -Wall -fno-semantic-interposition -msse4.2 -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized
-- CMAKE_CXX_FLAGS: -Wno-noexcept-type -Wall -fno-semantic-interposition -msse4.2 -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized
-- Generator: Unix Makefiles
-- Build output directory: /home/tom/Documents/dev/arrow/python/build/release
CMake Error at /home/tom/.local/lib/python3.8/site-packages/cmake/data/share/cmake-3.27/Modules/CMakeFindDependencyMacro.cmake:76 (find_package):
By not providing "FindOpenSSLAlt.cmake" in CMAKE_MODULE_PATH this project
has asked CMake to find a package configuration file provided by
"OpenSSLAlt", but CMake did not find one.
Could not find a package configuration file provided by "OpenSSLAlt" with
any of the following names:
OpenSSLAltConfig.cmake
opensslalt-config.cmake
Add the installation prefix of "OpenSSLAlt" to CMAKE_PREFIX_PATH or set
"OpenSSLAlt_DIR" to a directory containing one of the above files. If
"OpenSSLAlt" provides a separate development package or SDK, be sure it has
been installed.
Call Stack (most recent call first):
/home/tom/Documents/dev/arrow/cpp/build/src/arrow/ArrowConfig.cmake:99 (find_dependency)
/home/tom/Documents/dev/arrow/cpp/build/src/arrow/ArrowConfig.cmake:122 (arrow_find_dependencies)
CMakeLists.txt:268 (find_package)
-- Configuring incomplete, errors occurred!
error: command '/home/tom/.local/bin/cmake' failed with exit code 1
arrow/python on main [!?] via △ v3.27.9 via 🐍 v3.8.10 on ☁️ octopize (eu-west-3) on ☁️ tom@octopize.io
❯ python setup.py clean
running clean
removing 'build/temp.linux-x86_64-cpython-38' (and everything under it)
arrow/python on main [!?] via △ v3.27.9 via 🐍 v3.8.10 on ☁️ octopize (eu-west-3) on ☁️ tom@octopize.io
❯ PYARROW_CMAKE_OPTIONS="-DCMAKE_PREFIX_PATH=/home/tom/Documents/dev/dist" python setup.py build_ext --inplace
running build_ext
creating /home/tom/Documents/dev/arrow/python/build/temp.linux-x86_64-cpython-38
-- Running cmake for PyArrow
cmake -DCMAKE_PREFIX_PATH=/home/tom/Documents/dev/dist -DCMAKE_INSTALL_PREFIX=/home/tom/Documents/dev/arrow/python/pyarrow -DPYTHON_EXECUTABLE=/usr/bin/python -DPython3_EXECUTABLE=/usr/bin/python -DPYARROW_CXXFLAGS= -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_SUBSTRAIT=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ACERO=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PARQUET_ENCRYPTION=off -DPYARROW_BUILD_GCS=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_CYTHON_CPP=off -DPYARROW_GENERATE_COVERAGE=off -DCMAKE_BUILD_TYPE=release /home/tom/Documents/dev/arrow/python
-- System processor: x86_64
-- Arrow build warning level: PRODUCTION
-- Using ld linker
-- Build Type: RELEASE
-- CMAKE_C_FLAGS: -Wall -fno-semantic-interposition -msse4.2 -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized
-- CMAKE_CXX_FLAGS: -Wno-noexcept-type -Wall -fno-semantic-interposition -msse4.2 -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized
-- Generator: Unix Makefiles
-- Build output directory: /home/tom/Documents/dev/arrow/python/build/release
CMake Error at /home/tom/.local/lib/python3.8/site-packages/cmake/data/share/cmake-3.27/Modules/CMakeFindDependencyMacro.cmake:76 (find_package):
By not providing "FindOpenSSLAlt.cmake" in CMAKE_MODULE_PATH this project
has asked CMake to find a package configuration file provided by
"OpenSSLAlt", but CMake did not find one.
Could not find a package configuration file provided by "OpenSSLAlt" with
any of the following names:
OpenSSLAltConfig.cmake
opensslalt-config.cmake
Add the installation prefix of "OpenSSLAlt" to CMAKE_PREFIX_PATH or set
"OpenSSLAlt_DIR" to a directory containing one of the above files. If
"OpenSSLAlt" provides a separate development package or SDK, be sure it has
been installed.
Call Stack (most recent call first):
/home/tom/Documents/dev/arrow/cpp/build/src/arrow/ArrowConfig.cmake:99 (find_dependency)
/home/tom/Documents/dev/arrow/cpp/build/src/arrow/ArrowConfig.cmake:122 (arrow_find_dependencies)
CMakeLists.txt:268 (find_package)
-- Configuring incomplete, errors occurred!
error: command '/home/tom/.local/bin/cmake' failed with exit code 1
Hmm. Could you also try PYARROW_CMAKE_OPTIONS="-DCMAKE_PREFIX_PATH=/home/tom/Documents/dev/dist -DCMAKE_FIND_DEBUG_MODE=ON" python setup.py build_ext --inplace
?
It will show many logs. So attaching instead of pasting as a comment is better.
Here it is: pyarrow-build.log
Thanks but I couldn't find why /home/tom/Documents/dev/arrow/cpp/build/src/arrow/ArrowConfig.cmake
was used...
Could you show the output of env | sort | grep cpp/build
?
Returns nothing unfortunately.
Could you share some documentation around the tools that are used that I could read (I guess cmake
?), so that I can help debug and find the root cause?
I deleted arrow/cpp/build
, cd arrow/cpp && mkdir build
, executed all the same commands again, making sure to create a new dist_new
folder, in case there were some old paths laying around in cpp/build
or somewhere else, but still the same issue.
I did notice something though.
The error message says:
By not providing "FindOpenSSLAlt.cmake" in CMAKE_MODULE_PATH this project
has asked CMake to find a package configuration file provided by
"OpenSSLAlt", but CMake did not find one.
Though I do have it here:
❯ ls /home/tom/Documents/dev/dist_new/lib/cmake/Arrow/FindOpenSSLAlt.cmake
/home/tom/Documents/dev/dist_new/lib/cmake/Arrow/FindOpenSSLAlt.cmake
I tried: (notice -DCMAKE_MODULE_PATH=
)
❯ PYARROW_CMAKE_OPTIONS="-DCMAKE_PREFIX_PATH=/home/tom/Documents/dev/dist_new -DCMAKE_FIND_DEBUG_MODE=ON -DCMAKE_MODULE_PATH=/home/tom/Documents/dev/dist_new/lib/cmake/Arrow/" python setup.py build_ext --inplace
with no luck.
Log: build.log
But I guess that stems from the fact that the wrong ArrowConfig.cmake
is used.
I will check out the documentation you provided, thank you
Setting -DArrow_DIR=/home/tom/Documents/dev/dist_new/lib/cmake/Arrow/
made find_packge(Arrow)
look there first, and it worked!
I had to do the same for a few more packages, but I ended up being able to build it.
-DCMAKE_FIND_PACKAGE_REDIRECTS_DIR=/home/tom/Documents/dev/dist_new/lib/cmake/
should have been enough, but it didn't work.
Final command:
PYARROW_CMAKE_OPTIONS="-DArrowDataset_DIR=/home/tom/Documents/dev/dist_new/lib/cmake/ArrowDataset -DArrowAcero_DIR=/home/tom/Documents/dev/dist_new/lib/cmake/ArrowAcero -DParquet_DIR=/home/tom/Documents/dev/dist_new/lib/cmake/Parquet/ -DCMAKE_FIND_DEBUG_MODE=ON -DArrow_DIR=/home/tom/Documents/dev/dist_new/lib/cmake/Arrow/" python setup.py build_ext --inplace
Also, now removing these variables still makes it work afterwards. Do you know how I can clean the cache (if there is one)? I want to be sure that this is what led me further.
I tried:
-python setup.py clean
-git clean -Xfd python
rm -rf arrow/python/build
to no avail.
strace -f python setup.py build_ext --inplace
may help us.
Note that it outputs many logs because strace
outputs system call's trace logs. They have scanned paths. They may help us.)
No update. Can we close this as stale?
Yes, I used the examples/minimal_build Dockerfile as base in the end. Thanks for the help.
-------- Original Message -------- On 05/04/2024 09:52, Sutou Kouhei wrote:
No update. Can we close this as stale?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>
Describe the bug, including details regarding any error messages, version, and platform.
Platform:
I tried building Arrow C++ and
pyarrow
to start contributing, but I ran into an issue that I am not familiar with and that I could not debug myself.I followed the tutorial from the docs: https://arrow.apache.org/docs/developers/python.html#using-system-and-bundled-dependencies
Here are the exact commands I executed.
The output is:
The logs say that it did find OpenSSL:
but after building and installing (logs at the end of this post), and running the following commands to build
pyarrow
, I get the following error:I have tried to find the
OpenSSLAltConfig.cmake
file, but I can't find it anywhere:make install
logs:Component(s)
Packaging, Python