search

apache / arrow

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics
https://arrow.apache.org/
Apache License 2.0
14.6k stars 3.54k forks source link

[C++] How to build Apache Arrow C++ as dependency in CMake through CPM #43668

Open aosterthun opened 3 months ago

aosterthun commented 3 months ago

Describe the usage question you have. Please include as many useful details as possible.

I would like to build the Apache Arrow C++ library as part of my project. I use CPM to manage dependencies inside my project. I already got some help from the CPM community on how to set the source directory for the build since the CMakeLists file for Apache Arrow C++ is not in the root directory of the repository. While I am now able to actually download and configure Apache Arrow C++ as part of my projects CMake configuration step, I now get a CMake error (from the Apache Arrow C++ configuration) when trying to configure the project.

This is how I add Apache Arrow C++ to my project inside my CMakeLists file:

cpmaddpackage(
    NAME ApacheArrow
    GITHUB_REPOSITORY apache/arrow
    GIT_TAG apache-arrow-17.0.0
    SOURCE_SUBDIR "./cpp" # this is relative to the top directoy of the cloned repository
)

This is the CMake output I get when I try to configure my project:

[main] Configuring project: my_project 
[proc] Executing command: /usr/bin/cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_COMPILER=gcc-14 -DCMAKE_CXX_COMPILER=g++-14 -DCMAKE_EXPORT_COMPILE_COMMANDS=True -S/workspaces/my_project -B/workspaces/my_project/build/debug -G Ninja
[cmake] -- CPM: Adding package spdlog@1.12.0 (v1.12.0)
[cmake] -- Build spdlog: 1.12.0
[cmake] -- Build type: Debug
[cmake] -- CPM: Adding package Catch2@3.6.0 (v3.6.0)
[cmake] -- CPM: Adding package cxxopts@3.2.1 (v3.2.1)
[cmake] -- CPM: Adding package ApacheArrow@ (apache-arrow-17.0.0)
[cmake] -- Building using CMake version: 3.29.6
[cmake] -- Arrow version: 17.0.0 (full: '17.0.0')
[cmake] -- Arrow SO version: 1700 (full: 1700.0.0)
[cmake] -- clang-tidy 14 not found
[cmake] -- clang-format 14 not found
[cmake] -- Could NOT find ClangTools (missing: CLANG_FORMAT_BIN CLANG_TIDY_BIN) 
[cmake] -- infer not found
[cmake] -- Found cpplint executable at CPPLINT_BIN-NOTFOUND
[cmake] -- System processor: x86_64
[cmake] -- Arrow build warning level: CHECKIN
[cmake] CMake Warning at build/debug/_deps/apachearrow-src/cpp/cmake_modules/SetupCxxFlags.cmake:511 (message):
[cmake]   ARROW_SIMD_LEVEL= not supported by x86.
[cmake] Call Stack (most recent call first):
[cmake]   build/debug/_deps/apachearrow-src/cpp/CMakeLists.txt:446 (include)
[cmake] 
[cmake] 
[cmake] -- Using ld linker
[cmake] -- Build Type: DEBUG
[cmake] -- Using  approach to find dependencies
[cmake] -- ARROW_ABSL_BUILD_VERSION: 20211102.0
[cmake] -- ARROW_ABSL_BUILD_SHA256_CHECKSUM: dcf71b9cba8dc0ca9940c4b316a0c796be8fab42b070bb6b7cab62b48f0e66c4
[cmake] -- ARROW_AWS_C_AUTH_BUILD_VERSION: v0.6.22
[cmake] -- ARROW_AWS_C_AUTH_BUILD_SHA256_CHECKSUM: 691a6b4418afcd3dc141351b6ad33fccd8e3ff84df0e9e045b42295d284ee14c
[cmake] -- ARROW_AWS_C_CAL_BUILD_VERSION: v0.5.20
[cmake] -- ARROW_AWS_C_CAL_BUILD_SHA256_CHECKSUM: acc352359bd06f8597415c366cf4ec4f00d0b0da92d637039a73323dd55b6cd0
[cmake] -- ARROW_AWS_C_COMMON_BUILD_VERSION: v0.8.9
[cmake] -- ARROW_AWS_C_COMMON_BUILD_SHA256_CHECKSUM: 2f3fbaf7c38eae5a00e2a816d09b81177f93529ae8ba1b82dc8f31407565327a
[cmake] -- ARROW_AWS_C_COMPRESSION_BUILD_VERSION: v0.2.16
[cmake] -- ARROW_AWS_C_COMPRESSION_BUILD_SHA256_CHECKSUM: 044b1dbbca431a07bde8255ef9ec443c300fc60d4c9408d4b862f65e496687f4
[cmake] -- ARROW_AWS_C_EVENT_STREAM_BUILD_VERSION: v0.2.18
[cmake] -- ARROW_AWS_C_EVENT_STREAM_BUILD_SHA256_CHECKSUM: 310ca617f713bf664e4c7485a3d42c1fb57813abd0107e49790d107def7cde4f
[cmake] -- ARROW_AWS_C_HTTP_BUILD_VERSION: v0.7.3
[cmake] -- ARROW_AWS_C_HTTP_BUILD_SHA256_CHECKSUM: 07e16c6bf5eba6f0dea96b6f55eae312a7c95b736f4d2e4a210000f45d8265ae
[cmake] -- ARROW_AWS_C_IO_BUILD_VERSION: v0.13.14
[cmake] -- ARROW_AWS_C_IO_BUILD_SHA256_CHECKSUM: 12b66510c3d9a4f7e9b714e9cfab2a5bf835f8b9ce2f909d20ae2a2128608c71
[cmake] -- ARROW_AWS_C_MQTT_BUILD_VERSION: v0.8.4
[cmake] -- ARROW_AWS_C_MQTT_BUILD_SHA256_CHECKSUM: 232eeac63e72883d460c686a09b98cdd811d24579affac47c5c3f696f956773f
[cmake] -- ARROW_AWS_C_S3_BUILD_VERSION: v0.2.3
[cmake] -- ARROW_AWS_C_S3_BUILD_SHA256_CHECKSUM: a00b3c9f319cd1c9aa2c3fa15098864df94b066dcba0deaccbb3caa952d902fe
[cmake] -- ARROW_AWS_C_SDKUTILS_BUILD_VERSION: v0.1.6
[cmake] -- ARROW_AWS_C_SDKUTILS_BUILD_SHA256_CHECKSUM: 8a2951344b2fb541eab1e9ca17c18a7fcbfd2aaff4cdd31d362d1fad96111b91
[cmake] -- ARROW_AWS_CHECKSUMS_BUILD_VERSION: v0.1.13
[cmake] -- ARROW_AWS_CHECKSUMS_BUILD_SHA256_CHECKSUM: 0f897686f1963253c5069a0e495b85c31635ba146cd3ac38cc2ea31eaf54694d
[cmake] -- ARROW_AWS_CRT_CPP_BUILD_VERSION: v0.18.16
[cmake] -- ARROW_AWS_CRT_CPP_BUILD_SHA256_CHECKSUM: 9e69bc1dc4b50871d1038aa9ff6ddeb4c9b28f7d6b5e5b1b69041ccf50a13483
[cmake] -- ARROW_AWS_LC_BUILD_VERSION: v1.3.0
[cmake] -- ARROW_AWS_LC_BUILD_SHA256_CHECKSUM: ae96a3567161552744fc0cae8b4d68ed88b1ec0f3d3c98700070115356da5a37
[cmake] -- ARROW_AWSSDK_BUILD_VERSION: 1.10.55
[cmake] -- ARROW_AWSSDK_BUILD_SHA256_CHECKSUM: 2d552fb1a84bef4a9b65e34aa7031851ed2aef5319e02cc6e4cb735c48aa30de
[cmake] -- ARROW_AZURE_SDK_BUILD_VERSION: azure-core_1.10.3
[cmake] -- ARROW_AZURE_SDK_BUILD_SHA256_CHECKSUM: dd624c2f86adf474d2d0a23066be6e27af9cbd7e3f8d9d8fd7bf981e884b7b48
[cmake] -- ARROW_BOOST_BUILD_VERSION: 1.81.0
[cmake] -- ARROW_BOOST_BUILD_SHA256_CHECKSUM: 9e0ffae35528c35f90468997bc8d99500bf179cbae355415a89a600c38e13574
[cmake] -- ARROW_BROTLI_BUILD_VERSION: v1.0.9
[cmake] -- ARROW_BROTLI_BUILD_SHA256_CHECKSUM: f9e8d81d0405ba66d181529af42a3354f838c939095ff99930da6aa9cdf6fe46
[cmake] -- ARROW_BZIP2_BUILD_VERSION: 1.0.8
[cmake] -- ARROW_BZIP2_BUILD_SHA256_CHECKSUM: ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269
[cmake] -- ARROW_CARES_BUILD_VERSION: 1.17.2
[cmake] -- ARROW_CARES_BUILD_SHA256_CHECKSUM: 4803c844ce20ce510ef0eb83f8ea41fa24ecaae9d280c468c582d2bb25b3913d
[cmake] -- ARROW_CRC32C_BUILD_VERSION: 1.1.2
[cmake] -- ARROW_CRC32C_BUILD_SHA256_CHECKSUM: ac07840513072b7fcebda6e821068aa04889018f24e10e46181068fb214d7e56
[cmake] -- ARROW_GBENCHMARK_BUILD_VERSION: v1.8.3
[cmake] -- ARROW_GBENCHMARK_BUILD_SHA256_CHECKSUM: 6bc180a57d23d4d9515519f92b0c83d61b05b5bab188961f36ac7b06b0d9e9ce
[cmake] -- ARROW_GFLAGS_BUILD_VERSION: v2.2.2
[cmake] -- ARROW_GFLAGS_BUILD_SHA256_CHECKSUM: 34af2f15cf7367513b352bdcd2493ab14ce43692d2dcd9dfc499492966c64dcf
[cmake] -- ARROW_GLOG_BUILD_VERSION: v0.5.0
[cmake] -- ARROW_GLOG_BUILD_SHA256_CHECKSUM: eede71f28371bf39aa69b45de23b329d37214016e2055269b3b5e7cfd40b59f5
[cmake] -- ARROW_GOOGLE_CLOUD_CPP_BUILD_VERSION: v2.22.0
[cmake] -- ARROW_GOOGLE_CLOUD_CPP_BUILD_SHA256_CHECKSUM: 0c68782e57959c82e0c81def805c01460a042c1aae0c2feee905acaa2a2dc9bf
[cmake] -- ARROW_GRPC_BUILD_VERSION: v1.46.3
[cmake] -- ARROW_GRPC_BUILD_SHA256_CHECKSUM: d6cbf22cb5007af71b61c6be316a79397469c58c82a942552a62e708bce60964
[cmake] -- ARROW_GTEST_BUILD_VERSION: 1.11.0
[cmake] -- ARROW_GTEST_BUILD_SHA256_CHECKSUM: b4870bf121ff7795ba20d20bcdd8627b8e088f2d1dab299a031c1034eddc93d5
[cmake] -- ARROW_JEMALLOC_BUILD_VERSION: 5.3.0
[cmake] -- ARROW_JEMALLOC_BUILD_SHA256_CHECKSUM: 2db82d1e7119df3e71b7640219b6dfe84789bc0537983c3b7ac4f7189aecfeaa
[cmake] -- ARROW_LZ4_BUILD_VERSION: v1.9.4
[cmake] -- ARROW_LZ4_BUILD_SHA256_CHECKSUM: 0b0e3aa07c8c063ddf40b082bdf7e37a1562bda40a0ff5272957f3e987e0e54b
[cmake] -- ARROW_MIMALLOC_BUILD_VERSION: v2.0.6
[cmake] -- ARROW_MIMALLOC_BUILD_SHA256_CHECKSUM: 9f05c94cc2b017ed13698834ac2a3567b6339a8bde27640df5a1581d49d05ce5
[cmake] -- ARROW_NLOHMANN_JSON_BUILD_VERSION: v3.10.5
[cmake] -- ARROW_NLOHMANN_JSON_BUILD_SHA256_CHECKSUM: 5daca6ca216495edf89d167f808d1d03c4a4d929cef7da5e10f135ae1540c7e4
[cmake] -- ARROW_OPENTELEMETRY_BUILD_VERSION: v1.13.0
[cmake] -- ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM: 7735cc56507149686e6019e06f588317099d4522480be5f38a2a09ec69af1706
[cmake] -- ARROW_OPENTELEMETRY_PROTO_BUILD_VERSION: v0.17.0
[cmake] -- ARROW_OPENTELEMETRY_PROTO_BUILD_SHA256_CHECKSUM: f269fbcb30e17b03caa1decd231ce826e59d7651c0f71c3b28eb5140b4bb5412
[cmake] -- ARROW_ORC_BUILD_VERSION: 2.0.1
[cmake] -- ARROW_ORC_BUILD_SHA256_CHECKSUM: 1ffac0228aa83f04a1b1cf2788a3af5953e82587ae3a77c41900e99f2557132d
[cmake] -- ARROW_PROTOBUF_BUILD_VERSION: v21.3
[cmake] -- ARROW_PROTOBUF_BUILD_SHA256_CHECKSUM: 2f723218f6cb709ae4cdc4fb5ed56a5951fc5d466f0128ce4c946b8c78c8c49f
[cmake] -- ARROW_RAPIDJSON_BUILD_VERSION: 232389d4f1012dddec4ef84861face2d2ba85709
[cmake] -- ARROW_RAPIDJSON_BUILD_SHA256_CHECKSUM: b9290a9a6d444c8e049bd589ab804e0ccf2b05dc5984a19ed5ae75d090064806
[cmake] -- ARROW_RE2_BUILD_VERSION: 2022-06-01
[cmake] -- ARROW_RE2_BUILD_SHA256_CHECKSUM: f89c61410a072e5cbcf8c27e3a778da7d6fd2f2b5b1445cd4f4508bee946ab0f
[cmake] -- ARROW_SNAPPY_BUILD_VERSION: 1.1.10
[cmake] -- ARROW_SNAPPY_BUILD_SHA256_CHECKSUM: 49d831bffcc5f3d01482340fe5af59852ca2fe76c3e05df0e67203ebbe0f1d90
[cmake] -- ARROW_SUBSTRAIT_BUILD_VERSION: v0.44.0
[cmake] -- ARROW_SUBSTRAIT_BUILD_SHA256_CHECKSUM: f989a862f694e7dbb695925ddb7c4ce06aa6c51aca945105c075139aed7e55a2
[cmake] -- ARROW_S2N_TLS_BUILD_VERSION: v1.3.35
[cmake] -- ARROW_S2N_TLS_BUILD_SHA256_CHECKSUM: 9d32b26e6bfcc058d98248bf8fc231537e347395dd89cf62bb432b55c5da990d
[cmake] -- ARROW_THRIFT_BUILD_VERSION: 0.16.0
[cmake] -- ARROW_THRIFT_BUILD_SHA256_CHECKSUM: f460b5c1ca30d8918ff95ea3eb6291b3951cf518553566088f3f2be8981f6209
[cmake] -- ARROW_UCX_BUILD_VERSION: 1.12.1
[cmake] -- ARROW_UCX_BUILD_SHA256_CHECKSUM: 9bef31aed0e28bf1973d28d74d9ac4f8926c43ca3b7010bd22a084e164e31b71
[cmake] -- ARROW_UTF8PROC_BUILD_VERSION: v2.7.0
[cmake] -- ARROW_UTF8PROC_BUILD_SHA256_CHECKSUM: 4bb121e297293c0fd55f08f83afab6d35d48f0af4ecc07523ad8ec99aa2b12a1
[cmake] -- ARROW_XSIMD_BUILD_VERSION: 13.0.0
[cmake] -- ARROW_XSIMD_BUILD_SHA256_CHECKSUM: 8bdbbad0c3e7afa38d88d0d484d70a1671a1d8aefff03f4223ab2eb6a41110a3
[cmake] -- ARROW_ZLIB_BUILD_VERSION: 1.3.1
[cmake] -- ARROW_ZLIB_BUILD_SHA256_CHECKSUM: 9a93b2b7dfdac77ceba5a558a580e74667dd6fede4585b91eefb60f03b72df23
[cmake] -- ARROW_ZSTD_BUILD_VERSION: 1.5.6
[cmake] -- ARROW_ZSTD_BUILD_SHA256_CHECKSUM: 8c29e06cf42aacc1eafc4077ae2ec6c6fcb96a626157e0593d5e82a34fd403c1
[cmake] -- Building without OpenSSL support. Minimum OpenSSL version 1.0.2 required.
[cmake] -- xsimd found. Headers: 
[cmake] -- Found hdfs.h at: /workspaces/my_project/build/debug/_deps/apachearrow-src/cpp/thirdparty/hadoop/include/hdfs.h
[cmake] -- All bundled static libraries: 
[cmake] -- CMAKE_C_FLAGS:   -Wall -Wno-conversion -Wno-sign-conversion -Wdate-time -Wunused-result -fno-semantic-interposition 
[cmake] -- CMAKE_CXX_FLAGS:  -Wredundant-move -Wno-noexcept-type -Wno-self-move  -fdiagnostics-color=always  -Wall -Wno-conversion -Wno-sign-conversion -Wdate-time -Wunused-result -fno-semantic-interposition 
[cmake] -- CMAKE_C_FLAGS_DEBUG: -g -Werror -O0 
[cmake] -- CMAKE_CXX_FLAGS_DEBUG: -g -Werror -O0 
[cmake] CMake Error at build/debug/_deps/apachearrow-src/cpp/src/arrow/CMakeLists.txt:1090 (add_dependencies):
[cmake]   add_dependencies called with incorrect number of arguments
[cmake] 
[cmake] 
[cmake] CMake Error at build/debug/_deps/apachearrow-src/cpp/src/arrow/io/CMakeLists.txt:40 (if):
[cmake]   if given arguments:
[cmake] 
[cmake]     "NOT" "(" "STREQUAL" "NONE" ")" "AND" "NOT" "(" "STREQUAL" "NEON" ")"
[cmake] 
[cmake]   Unknown arguments specified
[cmake] 
[cmake] 
[cmake] -- Configuring incomplete, errors occurred!
[proc] The command: /usr/bin/cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_COMPILER=gcc-14 -DCMAKE_CXX_COMPILER=g++-14 -DCMAKE_EXPORT_COMPILE_COMMANDS=True -S/workspaces/my_project -B/workspaces/my_project/build/debug -G Ninja exited with code: 1

I am able to build Apache Arrow C++ as a standalone project, so there must be some interaction with CPM (Which is build on top of FetchContent).

Component(s)

C++

bkietz commented 3 months ago

Hi! Our options resolution (see DefineOptions.cmake) isn't currently equipped to deal with CPM and our dependency management isn't currently aware of CPM (see ThirdpartyToolchain.cmake::resolve_dependency()). To get a minimal build working, pass some cmake options explicitly:

cmake_minimum_required(VERSION 3.30)
project(arrow_with_CPM)
include(cmake/CPM.cmake)
cpmaddpackage(
    NAME ApacheArrow
    GITHUB_REPOSITORY apache/arrow
    GIT_TAG apache-arrow-17.0.0
    SOURCE_SUBDIR "./cpp" # this is relative to the top directoy of the cloned repository
    OPTIONS
      "ARROW_DEFINE_OPTIONS ON"
      "ARROW_SIMD_LEVEL NONE"
      "ARROW_RUNTIME_SIMD_LEVEL NONE"
)

In the longer term, it'd be neat to support acquiring our own dependencies with CMake

kou commented 3 months ago

It seems that we can use Apache Arrow C++ with CPM by adding support for FetchContent.

aosterthun commented 3 months ago

Hi! Our options resolution (see DefineOptions.cmake) isn't currently equipped to deal with CPM and our dependency management isn't currently aware of CPM (see ThirdpartyToolchain.cmake::resolve_dependency()). To get a minimal build working, pass some cmake options explicitly:

cmake_minimum_required(VERSION 3.30)
project(arrow_with_CPM)
include(cmake/CPM.cmake)
cpmaddpackage(
    NAME ApacheArrow
    GITHUB_REPOSITORY apache/arrow
    GIT_TAG apache-arrow-17.0.0
    SOURCE_SUBDIR "./cpp" # this is relative to the top directoy of the cloned repository
    OPTIONS
      "ARROW_DEFINE_OPTIONS ON"
      "ARROW_SIMD_LEVEL NONE"
      "ARROW_RUNTIME_SIMD_LEVEL NONE"
)

In the longer term, it'd be neat to support acquiring our own dependencies with CMake

Thank you, this works seamlessly for the most part. I only had to change the target names from Parquet::parquet_shared and Arrow::arrow_shared to parquet_shared and arrow_shared when linking them against my executable.

As already mentioned it would be convenient to have more usable default options, but for now I am happy this works at all!

ligazetom commented 2 months ago

Hey I'm trying to build only the PARQUET module using these settings.

CPMDeclarePackage(apache-arrow
  NAME apache-arrow
  GIT_TAG apache-arrow-17.0.0
  GITHUB_REPOSITORY apache/arrow
  SOURCE_SUBDIR "cpp"
  OPTIONS
    "ARROW_DEFINE_OPTIONS ON"
    "ARROW_SIMD_LEVEL NONE"
    "ARROW_RUNTIME_SIMD_LEVEL NONE"
    "ARROW_PARQUET ON"
  )

And I'm getting this output.

[cmake] -- CPM: Adding package apache-arrow@ (apache-arrow-17.0.0)
[cmake] -- Building using CMake version: 3.22.1
[cmake] -- Arrow version: 17.0.0 (full: '17.0.0')
[cmake] -- Arrow SO version: 1700 (full: 1700.0.0)
[cmake] -- clang-tidy 14 not found
[cmake] -- clang-format found, but version did not match "clang-format version 14"
[cmake] -- clang-format 14 not found
[cmake] -- Could NOT find ClangTools (missing: CLANG_FORMAT_BIN CLANG_TIDY_BIN) 
[cmake] -- infer not found
[cmake] -- Found cpplint executable at CPPLINT_BIN-NOTFOUND
[cmake] -- System processor: aarch64
[cmake] -- Arrow build warning level: CHECKIN
[cmake] -- Using ld linker
[cmake] -- Build Type: DEBUG
[cmake] -- Using AUTO approach to find dependencies
[cmake] -- ARROW_ABSL_BUILD_VERSION: 20211102.0
[cmake] -- ARROW_ABSL_BUILD_SHA256_CHECKSUM: dcf71b9cba8dc0ca9940c4b316a0c796be8fab42b070bb6b7cab62b48f0e66c4
[cmake] -- ARROW_AWS_C_AUTH_BUILD_VERSION: v0.6.22
[cmake] -- ARROW_AWS_C_AUTH_BUILD_SHA256_CHECKSUM: 691a6b4418afcd3dc141351b6ad33fccd8e3ff84df0e9e045b42295d284ee14c
[cmake] -- ARROW_AWS_C_CAL_BUILD_VERSION: v0.5.20
[cmake] -- ARROW_AWS_C_CAL_BUILD_SHA256_CHECKSUM: acc352359bd06f8597415c366cf4ec4f00d0b0da92d637039a73323dd55b6cd0
[cmake] -- ARROW_AWS_C_COMMON_BUILD_VERSION: v0.8.9
[cmake] -- ARROW_AWS_C_COMMON_BUILD_SHA256_CHECKSUM: 2f3fbaf7c38eae5a00e2a816d09b81177f93529ae8ba1b82dc8f31407565327a
[cmake] -- ARROW_AWS_C_COMPRESSION_BUILD_VERSION: v0.2.16
[cmake] -- ARROW_AWS_C_COMPRESSION_BUILD_SHA256_CHECKSUM: 044b1dbbca431a07bde8255ef9ec443c300fc60d4c9408d4b862f65e496687f4
[cmake] -- ARROW_AWS_C_EVENT_STREAM_BUILD_VERSION: v0.2.18
[cmake] -- ARROW_AWS_C_EVENT_STREAM_BUILD_SHA256_CHECKSUM: 310ca617f713bf664e4c7485a3d42c1fb57813abd0107e49790d107def7cde4f
[cmake] -- ARROW_AWS_C_HTTP_BUILD_VERSION: v0.7.3
[cmake] -- ARROW_AWS_C_HTTP_BUILD_SHA256_CHECKSUM: 07e16c6bf5eba6f0dea96b6f55eae312a7c95b736f4d2e4a210000f45d8265ae
[cmake] -- ARROW_AWS_C_IO_BUILD_VERSION: v0.13.14
[cmake] -- ARROW_AWS_C_IO_BUILD_SHA256_CHECKSUM: 12b66510c3d9a4f7e9b714e9cfab2a5bf835f8b9ce2f909d20ae2a2128608c71
[cmake] -- ARROW_AWS_C_MQTT_BUILD_VERSION: v0.8.4
[cmake] -- ARROW_AWS_C_MQTT_BUILD_SHA256_CHECKSUM: 232eeac63e72883d460c686a09b98cdd811d24579affac47c5c3f696f956773f
[cmake] -- ARROW_AWS_C_S3_BUILD_VERSION: v0.2.3
[cmake] -- ARROW_AWS_C_S3_BUILD_SHA256_CHECKSUM: a00b3c9f319cd1c9aa2c3fa15098864df94b066dcba0deaccbb3caa952d902fe
[cmake] -- ARROW_AWS_C_SDKUTILS_BUILD_VERSION: v0.1.6
[cmake] -- ARROW_AWS_C_SDKUTILS_BUILD_SHA256_CHECKSUM: 8a2951344b2fb541eab1e9ca17c18a7fcbfd2aaff4cdd31d362d1fad96111b91
[cmake] -- ARROW_AWS_CHECKSUMS_BUILD_VERSION: v0.1.13
[cmake] -- ARROW_AWS_CHECKSUMS_BUILD_SHA256_CHECKSUM: 0f897686f1963253c5069a0e495b85c31635ba146cd3ac38cc2ea31eaf54694d
[cmake] -- ARROW_AWS_CRT_CPP_BUILD_VERSION: v0.18.16
[cmake] -- ARROW_AWS_CRT_CPP_BUILD_SHA256_CHECKSUM: 9e69bc1dc4b50871d1038aa9ff6ddeb4c9b28f7d6b5e5b1b69041ccf50a13483
[cmake] -- ARROW_AWS_LC_BUILD_VERSION: v1.3.0
[cmake] -- ARROW_AWS_LC_BUILD_SHA256_CHECKSUM: ae96a3567161552744fc0cae8b4d68ed88b1ec0f3d3c98700070115356da5a37
[cmake] -- ARROW_AWSSDK_BUILD_VERSION: 1.10.55
[cmake] -- ARROW_AWSSDK_BUILD_SHA256_CHECKSUM: 2d552fb1a84bef4a9b65e34aa7031851ed2aef5319e02cc6e4cb735c48aa30de
[cmake] -- ARROW_AZURE_SDK_BUILD_VERSION: azure-core_1.10.3
[cmake] -- ARROW_AZURE_SDK_BUILD_SHA256_CHECKSUM: dd624c2f86adf474d2d0a23066be6e27af9cbd7e3f8d9d8fd7bf981e884b7b48
[cmake] -- ARROW_BOOST_BUILD_VERSION: 1.81.0
[cmake] -- ARROW_BOOST_BUILD_SHA256_CHECKSUM: 9e0ffae35528c35f90468997bc8d99500bf179cbae355415a89a600c38e13574
[cmake] -- ARROW_BROTLI_BUILD_VERSION: v1.0.9
[cmake] -- ARROW_BROTLI_BUILD_SHA256_CHECKSUM: f9e8d81d0405ba66d181529af42a3354f838c939095ff99930da6aa9cdf6fe46
[cmake] -- ARROW_BZIP2_BUILD_VERSION: 1.0.8
[cmake] -- ARROW_BZIP2_BUILD_SHA256_CHECKSUM: ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269
[cmake] -- ARROW_CARES_BUILD_VERSION: 1.17.2
[cmake] -- ARROW_CARES_BUILD_SHA256_CHECKSUM: 4803c844ce20ce510ef0eb83f8ea41fa24ecaae9d280c468c582d2bb25b3913d
[cmake] -- ARROW_CRC32C_BUILD_VERSION: 1.1.2
[cmake] -- ARROW_CRC32C_BUILD_SHA256_CHECKSUM: ac07840513072b7fcebda6e821068aa04889018f24e10e46181068fb214d7e56
[cmake] -- ARROW_GBENCHMARK_BUILD_VERSION: v1.8.3
[cmake] -- ARROW_GBENCHMARK_BUILD_SHA256_CHECKSUM: 6bc180a57d23d4d9515519f92b0c83d61b05b5bab188961f36ac7b06b0d9e9ce
[cmake] -- ARROW_GFLAGS_BUILD_VERSION: v2.2.2
[cmake] -- ARROW_GFLAGS_BUILD_SHA256_CHECKSUM: 34af2f15cf7367513b352bdcd2493ab14ce43692d2dcd9dfc499492966c64dcf
[cmake] -- ARROW_GLOG_BUILD_VERSION: v0.5.0
[cmake] -- ARROW_GLOG_BUILD_SHA256_CHECKSUM: eede71f28371bf39aa69b45de23b329d37214016e2055269b3b5e7cfd40b59f5
[cmake] -- ARROW_GOOGLE_CLOUD_CPP_BUILD_VERSION: v2.22.0
[cmake] -- ARROW_GOOGLE_CLOUD_CPP_BUILD_SHA256_CHECKSUM: 0c68782e57959c82e0c81def805c01460a042c1aae0c2feee905acaa2a2dc9bf
[cmake] -- ARROW_GRPC_BUILD_VERSION: v1.46.3
[cmake] -- ARROW_GRPC_BUILD_SHA256_CHECKSUM: d6cbf22cb5007af71b61c6be316a79397469c58c82a942552a62e708bce60964
[cmake] -- ARROW_GTEST_BUILD_VERSION: 1.11.0
[cmake] -- ARROW_GTEST_BUILD_SHA256_CHECKSUM: b4870bf121ff7795ba20d20bcdd8627b8e088f2d1dab299a031c1034eddc93d5
[cmake] -- ARROW_JEMALLOC_BUILD_VERSION: 5.3.0
[cmake] -- ARROW_JEMALLOC_BUILD_SHA256_CHECKSUM: 2db82d1e7119df3e71b7640219b6dfe84789bc0537983c3b7ac4f7189aecfeaa
[cmake] -- ARROW_LZ4_BUILD_VERSION: v1.9.4
[cmake] -- ARROW_LZ4_BUILD_SHA256_CHECKSUM: 0b0e3aa07c8c063ddf40b082bdf7e37a1562bda40a0ff5272957f3e987e0e54b
[cmake] -- ARROW_MIMALLOC_BUILD_VERSION: v2.0.6
[cmake] -- ARROW_MIMALLOC_BUILD_SHA256_CHECKSUM: 9f05c94cc2b017ed13698834ac2a3567b6339a8bde27640df5a1581d49d05ce5
[cmake] -- ARROW_NLOHMANN_JSON_BUILD_VERSION: v3.10.5
[cmake] -- ARROW_NLOHMANN_JSON_BUILD_SHA256_CHECKSUM: 5daca6ca216495edf89d167f808d1d03c4a4d929cef7da5e10f135ae1540c7e4
[cmake] -- ARROW_OPENTELEMETRY_BUILD_VERSION: v1.13.0
[cmake] -- ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM: 7735cc56507149686e6019e06f588317099d4522480be5f38a2a09ec69af1706
[cmake] -- ARROW_OPENTELEMETRY_PROTO_BUILD_VERSION: v0.17.0
[cmake] -- ARROW_OPENTELEMETRY_PROTO_BUILD_SHA256_CHECKSUM: f269fbcb30e17b03caa1decd231ce826e59d7651c0f71c3b28eb5140b4bb5412
[cmake] -- ARROW_ORC_BUILD_VERSION: 2.0.1
[cmake] -- ARROW_ORC_BUILD_SHA256_CHECKSUM: 1ffac0228aa83f04a1b1cf2788a3af5953e82587ae3a77c41900e99f2557132d
[cmake] -- ARROW_PROTOBUF_BUILD_VERSION: v21.3
[cmake] -- ARROW_PROTOBUF_BUILD_SHA256_CHECKSUM: 2f723218f6cb709ae4cdc4fb5ed56a5951fc5d466f0128ce4c946b8c78c8c49f
[cmake] -- ARROW_RAPIDJSON_BUILD_VERSION: 232389d4f1012dddec4ef84861face2d2ba85709
[cmake] -- ARROW_RAPIDJSON_BUILD_SHA256_CHECKSUM: b9290a9a6d444c8e049bd589ab804e0ccf2b05dc5984a19ed5ae75d090064806
[cmake] -- ARROW_RE2_BUILD_VERSION: 2022-06-01
[cmake] -- ARROW_RE2_BUILD_SHA256_CHECKSUM: f89c61410a072e5cbcf8c27e3a778da7d6fd2f2b5b1445cd4f4508bee946ab0f
[cmake] -- ARROW_SNAPPY_BUILD_VERSION: 1.1.10
[cmake] -- ARROW_SNAPPY_BUILD_SHA256_CHECKSUM: 49d831bffcc5f3d01482340fe5af59852ca2fe76c3e05df0e67203ebbe0f1d90
[cmake] -- ARROW_SUBSTRAIT_BUILD_VERSION: v0.44.0
[cmake] -- ARROW_SUBSTRAIT_BUILD_SHA256_CHECKSUM: f989a862f694e7dbb695925ddb7c4ce06aa6c51aca945105c075139aed7e55a2
[cmake] -- ARROW_S2N_TLS_BUILD_VERSION: v1.3.35
[cmake] -- ARROW_S2N_TLS_BUILD_SHA256_CHECKSUM: 9d32b26e6bfcc058d98248bf8fc231537e347395dd89cf62bb432b55c5da990d
[cmake] -- ARROW_THRIFT_BUILD_VERSION: 0.16.0
[cmake] -- ARROW_THRIFT_BUILD_SHA256_CHECKSUM: f460b5c1ca30d8918ff95ea3eb6291b3951cf518553566088f3f2be8981f6209
[cmake] -- ARROW_UCX_BUILD_VERSION: 1.12.1
[cmake] -- ARROW_UCX_BUILD_SHA256_CHECKSUM: 9bef31aed0e28bf1973d28d74d9ac4f8926c43ca3b7010bd22a084e164e31b71
[cmake] -- ARROW_UTF8PROC_BUILD_VERSION: v2.7.0
[cmake] -- ARROW_UTF8PROC_BUILD_SHA256_CHECKSUM: 4bb121e297293c0fd55f08f83afab6d35d48f0af4ecc07523ad8ec99aa2b12a1
[cmake] -- ARROW_XSIMD_BUILD_VERSION: 13.0.0
[cmake] -- ARROW_XSIMD_BUILD_SHA256_CHECKSUM: 8bdbbad0c3e7afa38d88d0d484d70a1671a1d8aefff03f4223ab2eb6a41110a3
[cmake] -- ARROW_ZLIB_BUILD_VERSION: 1.3.1
[cmake] -- ARROW_ZLIB_BUILD_SHA256_CHECKSUM: 9a93b2b7dfdac77ceba5a558a580e74667dd6fede4585b91eefb60f03b72df23
[cmake] -- ARROW_ZSTD_BUILD_VERSION: 1.5.6
[cmake] -- ARROW_ZSTD_BUILD_SHA256_CHECKSUM: 8c29e06cf42aacc1eafc4077ae2ec6c6fcb96a626157e0593d5e82a34fd403c1
[cmake] -- Found Boost: /usr/lib/aarch64-linux-gnu/cmake/Boost-1.74.0/BoostConfig.cmake (found suitable version "1.74.0", minimum required is "1.58")  
[cmake] CMake Error at build/_deps/apache-arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:1313 (target_link_libraries):
[cmake]   Cannot specify link libraries for target "Boost::headers" which is not
[cmake]   built by this project.
[cmake] Call Stack (most recent call first):
[cmake]   build/_deps/apache-arrow-src/cpp/CMakeLists.txt:544 (include)

I've seen the Boost::headers issue in multiple places related to this. I've got libboost1.74-all-dev installed in my env and I can see cmake found it while configuring.

I also tried it with

CPMDeclarePackage(apache-arrow
  NAME apache-arrow
  GIT_TAG apache-arrow-17.0.0
  GITHUB_REPOSITORY apache/arrow
  SOURCE_SUBDIR "cpp"
  OPTIONS
    "ARROW_DEFINE_OPTIONS ON"
    "ARROW_SIMD_LEVEL NONE"
    "ARROW_RUNTIME_SIMD_LEVEL NONE"
    "ARROW_PARQUET ON"
    "PARQUET_MINIMAL_DEPENDENCY OFF"
    "ARROW_BUILD_TESTS ON"
    "Thrift_SOURCE BUNDLED"
    "BOOST_SOURCE BUNDLED"
    "ARROW_OPTIONAL_INSTALL OFF"
  )

based on some googling and it just got worse with

[cmake] CMake Error at build/_deps/apache-arrow-src/cpp/cmake_modules/BuildUtils.cmake:46 (add_library):
[cmake]   add_library cannot create imported target "Boost::system" because another
[cmake]   target with the same name already exists.
[cmake] Call Stack (most recent call first):
[cmake]   build/_deps/apache-arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:1130 (add_thirdparty_lib)
[cmake]   build/_deps/apache-arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:182 (build_boost)
[cmake]   build/_deps/apache-arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:308 (build_dependency)
[cmake]   build/_deps/apache-arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:1296 (resolve_dependency)
[cmake]   build/_deps/apache-arrow-src/cpp/CMakeLists.txt:544 (include)
[cmake] 
[cmake] 
[cmake] CMake Error at build/_deps/apache-arrow-src/cpp/cmake_modules/BuildUtils.cmake:46 (add_library):
[cmake]   add_library cannot create imported target "Boost::filesystem" because
[cmake]   another target with the same name already exists.
[cmake] Call Stack (most recent call first):
[cmake]   build/_deps/apache-arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:1135 (add_thirdparty_lib)
[cmake]   build/_deps/apache-arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:182 (build_boost)
[cmake]   build/_deps/apache-arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:308 (build_dependency)
[cmake]   build/_deps/apache-arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:1296 (resolve_dependency)
[cmake]   build/_deps/apache-arrow-src/cpp/CMakeLists.txt:544 (include)
[cmake] 
[cmake] 
[cmake] CMake Error at build/_deps/apache-arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:1161 (add_library):
[cmake]   add_library cannot create imported target "Boost::headers" because another
[cmake]   target with the same name already exists.
[cmake] Call Stack (most recent call first):
[cmake]   build/_deps/apache-arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:182 (build_boost)
[cmake]   build/_deps/apache-arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:308 (build_dependency)
[cmake]   build/_deps/apache-arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:1296 (resolve_dependency)
[cmake]   build/_deps/apache-arrow-src/cpp/CMakeLists.txt:544 (include)
[cmake] 
[cmake] 
[cmake] CMake Error at build/_deps/apache-arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:1313 (target_link_libraries):
[cmake]   Cannot specify link libraries for target "Boost::headers" which is not
[cmake]   built by this project.
[cmake] Call Stack (most recent call first):
[cmake]   build/_deps/apache-arrow-src/cpp/CMakeLists.txt:544 (include)

I'm not being able to build without ARROW_PARQUET ON because I have lint target as well somewhere within the project so it's completely different problem. But still I'd like to build only the parquet module as I don't need anything else. Please let me know If you have any ideas. I was able to build with bash manually.

Edit: I built manually using this

cmake .. -DARROW_PARQUET=ON \
      -DARROW_OPTIONAL_INSTALL=OFF \
      -DARROW_BUILD_TESTS=OFF
make parquet
make install
bkietz commented 2 months ago

@kou it seems we can't target_link_libraries to attach things to imported targets (as when system boost is being used). Do we have a CI job which exercises using system boost? This seems like system boost is always broken

[cmake] -- Found Boost: /usr/lib/aarch64-linux-gnu/cmake/Boost-1.74.0/BoostConfig.cmake (found suitable version "1.74.0", minimum required is "1.58")  
[cmake] CMake Error at build/_deps/apache-arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:1313 (target_link_libraries):
[cmake]   Cannot specify link libraries for target "Boost::headers" which is not
[cmake]   built by this project.

it just got worse with

This looks like the options were modified without deleting the build directory/--fresh/...

kou commented 2 months ago

Do we have a CI job which exercises using system boost?

Yes. Most CI jobs use system Boost.

For example:

https://github.com/apache/arrow/actions/runs/10745114977/job/29803486094#step:7:3939

-- Found Boost: /usr/lib/aarch64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake (found suitable version "1.71.0", minimum required is "1.64") found components: filesystem system missing components: process

https://github.com/apache/arrow/actions/runs/10745114977/job/29803485926#step:6:805

-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.74.0/BoostConfig.cmake (found suitable version "1.74.0", minimum required is "1.64") found components: filesystem system missing components: process

And I'm always use system Boost for my local build.

I think that we need to add support for FetchContent as I said before...

ligazetom commented 2 months ago

@bkietz

This looks like the options were modified without deleting the build directory/--fresh/...

Unfortunately no. Everything is for a clean clean build.

I'll just try to use it as part of the env for now hoping it will work sometimes later. Thank you.