Unable to install Arrow 17.0.0 with GCS enabled via conan 2.6.0

[ppkowalski]

Describe the bug, including details regarding any error messages, version, and platform.

I am trying to install arrow 17 with conan 2.6.0 on Ubuntu22.04. My conanfile.py:

from conan import ConanFile

class ArrowTest(ConanFile):
    name = "arrow test"
    license = "MIT"
    description = "Arrow test"
    options = {"shared": [True, False]}
    default_options = {
        "shared": False,
        "arrow/*:acero": True,
        "arrow/*:compute": True,
        "arrow/*:dataset_modules": True,
        "arrow/*:parquet": True,
        "arrow/*:with_boost": True,
        "arrow/*:with_thrift": True,
        "arrow/*:with_flight_rpc": True,
        "arrow/*:filesystem_layer": True,
        "arrow/*:with_protobuf": True,
        "arrow/*:with_gflags": True,
        "arrow/*:with_csv": True,
        "arrow/*:with_gcs": True,
        "arrow/*:with_re2": True,
        "arrow/*:with_grpc": True,
    }

    generators = ["CMakeDeps", "CMakeToolchain"]
    exports_sources = ["CMakeLists.txt", "src/*"]
    settings = "build_type"
    build_policy = "missing"

    def requirements(self):
        self.requires("arrow/17.0.0")
        self.requires("grpc/1.54.3", override=True)
        self.requires("google-cloud-cpp/2.12.0", override=True)

The overrides are needed, because google-cloud-cpp is requested by arrow in version 1.40.1 which is not supported by conan 2.

However at some point I am getting following error:

-- Providing CMake module for FindgRPCAlt as part of ArrowFlight CMake package
-- pkg-config package for grpc++ that is used by arrow-flight for static link isn't found
CMake Error at cmake_modules/ThirdpartyToolchain.cmake:310 (find_package):
  By not providing "Findnlohmann_json.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "nlohmann_json", but CMake did not find one.

  Could not find a package configuration file provided by "nlohmann_json"
  with any of the following names:

    nlohmann_jsonConfig.cmake
    nlohmann_json-config.cmake

  Add the installation prefix of "nlohmann_json" to CMAKE_PREFIX_PATH or set
  "nlohmann_json_DIR" to a directory containing one of the above files.  If
  "nlohmann_json" provides a separate development package or SDK, be sure it
  has been installed.
Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:4251 (resolve_dependency)
  CMakeLists.txt:544 (include)

-- Configuring incomplete, errors occurred!

arrow/17.0.0: ERROR:
Package 'e67d4e73024d64e3e32a44a09882bca88d817bbf' build failed
arrow/17.0.0: WARN: Build folder /home/ppkowalski/.conan2/p/b/arrow3de7586871a98/b/build/Debug
ERROR: arrow/17.0.0: Error in build() method, line 414
        cmake.configure(build_script_folder=os.path.join(self.source_folder, "cpp"))
        ConanException: Error 1 while executing

I thought about installing nlohmann-json3-dev package using apt on the system (which I prefer not to, since I'd like to install everything with conan), then it goes further, however I am getting another error:

-- Providing CMake module for FindgRPCAlt as part of ArrowFlight CMake package
-- pkg-config package for grpc++ that is used by arrow-flight for static link isn't found
-- Found nlohmann_json: /usr/lib/cmake/nlohmann_json/nlohmann_jsonConfig.cmake (found version "3.10.5")
-- Found nlohmann_json headers: /usr/include;/usr/include
CMake Error at cmake_modules/ThirdpartyToolchain.cmake:310 (find_package):
  By not providing "Findgoogle_cloud_cpp_storage.cmake" in CMAKE_MODULE_PATH
  this project has asked CMake to find a package configuration file provided
  by "google_cloud_cpp_storage", but CMake did not find one.

  Could not find a package configuration file provided by
  "google_cloud_cpp_storage" with any of the following names:

    google_cloud_cpp_storageConfig.cmake
    google_cloud_cpp_storage-config.cmake

  Add the installation prefix of "google_cloud_cpp_storage" to
  CMAKE_PREFIX_PATH or set "google_cloud_cpp_storage_DIR" to a directory
  containing one of the above files.  If "google_cloud_cpp_storage" provides
  a separate development package or SDK, be sure it has been installed.
Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:4462 (resolve_dependency)
  CMakeLists.txt:544 (include)

-- Configuring incomplete, errors occurred!

arrow/17.0.0: ERROR:
Package 'e67d4e73024d64e3e32a44a09882bca88d817bbf' build failed
arrow/17.0.0: WARN: Build folder /home/ppkowalski/.conan2/p/b/arrow191307691fc54/b/build/Debug
ERROR: arrow/17.0.0: Error in build() method, line 414
        cmake.configure(build_script_folder=os.path.join(self.source_folder, "cpp"))
        ConanException: Error 1 while executing

How can I proceed?

Component(s)

Packaging

[kou]

Could you use https://github.com/conan-io/conan-center-index/tree/master/recipes/nlohmann_json instead of nlohmann-json3-dev?

Could you share a list of installed files by google-cloud-cpp?

The overrides are needed, because google-cloud-cpp is requested by arrow in version 1.40.1 which is not supported by conan 2.

Could you report it to the upstream? https://github.com/conan-io/conan-center-index/blob/89d6c8edf50dbccefe84c24cfce45d449debfeb5/recipes/arrow/all/conanfile.py#L179

[ppkowalski]

Could you use https://github.com/conan-io/conan-center-index/tree/master/recipes/nlohmann_json instead of nlohmann-json3-dev?

I tried adding explicit dependency to nlohmann_json in my conanfile.py, but it did not help. The problem seems to be in cmake_modules/ThirdpartyToolchain.cmake which tries to install this dependency on its own (unless it is installed by apt). Similar problem applies to google libs, as they are being installed by conan, but ThirdpartyToolchain tries to download sources and compile it on its own

Could you report it to the upstream?

I did so: https://github.com/conan-io/conan-center-index/issues/25011

[kou]

Could you show full log? It's strange that ThirdpartyToolchain.cmake downloads and compile. It must not happen because ARROW_DEPENDENCY_SOURCE=SYSTEM is used: https://github.com/conan-io/conan-center-index/blob/a848a025cc67040b7d496cd31dc8a58125f160c4/recipes/arrow/all/conanfile.py#L284 With the configuration, it must fail when system dependencies aren't found.

[ppkowalski]

Please find following log files:

conan-json.log which shows arrow install attempt (after other dependencies has been installed) WITHOUT nlohmann-json installed in the system conan-json-from-cc.log with nlohmann-json/3.11.3 added to the conanfile.py and installed from conan center conan-google.log after running apt-get install nlohmann-json3-dev

[kou]

Thanks but they don't include information what I know...

Could you also show the followings?

find /home/ppkowalski/.conan2/p/b/nlohmann_json*

find /home/ppkowalski/.conan2/p/b/google-cloud-cpp*

BTW, the arrow conan recipe misses nlohmann_json as a build dependency with "arrow/*:with_gcs": True. Could you also report it to the upstream?

[ppkowalski]

Please find the files attached:

find /home/ppkowalski/.conan2/p/nlohm* since it's header only (nothing found in p/b) nlohmann_json.log

find /home/ppkowalski/.conan2/p/b/googl*/ google.log

[kou]

Thanks.

find /home/ppkowalski/.conan2/p/nlohm* since it's header only (nothing found in p/b) nlohmann_json.log

This may be a problem. Could you open a new issue to conan-center-index for adding support for CMake integration? See also: https://github.com/nlohmann/json?tab=readme-ov-file#cmake

find /home/ppkowalski/.conan2/p/b/googl*/ google.log

Hmm. Do you have installed /home/ppkowalski/.conan2/p/b/googl0544eb5499220/b/build/Debug/google/cloud/storage/google_cloud_cpp_storage-config.cmake? We need to locate it to ${PREFIX}/cmake/google_cloud_cpp_storage/google_cloud_cpp_storage-config.cmake. See also: https://cmake.org/cmake/help/latest/command/find_package.html#search-procedure