search

apache / arrow

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
https://arrow.apache.org/
Apache License 2.0
14.36k stars 3.49k forks source link

[CI][C++][Homebrew] Failed to detect Protobuf by ORC #42149

Closed kou closed 2 months ago

kou commented 3 months ago

Describe the bug, including details regarding any error messages, version, and platform.

ARM64 macOS 14 C++: https://github.com/apache/arrow/actions/runs/9508523884/job/26209983344#step:12:862

[25/1026] Performing configure step for 'orc_ep'
FAILED: orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure 
cd /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-build && /opt/homebrew/Cellar/cmake/3.29.5/bin/cmake -P /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure-DEBUG.cmake && /opt/homebrew/Cellar/cmake/3.29.5/bin/cmake -E touch /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure
CMake Error at /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure-DEBUG.cmake:37 (message):
  Command failed: 1

   '/opt/homebrew/Cellar/cmake/3.29.5/bin/cmake' '-DCMAKE_C_COMPILER=/Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc' '-DCMAKE_CXX_COMPILER=/Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++' '-DCMAKE_AR=/Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar' '-DCMAKE_RANLIB=/Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib' '-DBUILD_SHARED_LIBS=OFF' '-DBUILD_STATIC_LIBS=ON' '-DBUILD_TESTING=OFF' '-DCMAKE_BUILD_TYPE=DEBUG' '-DCMAKE_CXX_FLAGS= -Qunused-arguments -fcolor-diagnostics -fPIC' '-DCMAKE_CXX_FLAGS_DEBUG=-g -Werror -O0 -ggdb -g1  -Wno-error' '-DCMAKE_CXX_FLAGS_MISIZEREL=-Os -DNDEBUG ' '-DCMAKE_CXX_FLAGS_RELEASE=-O3 -DNDEBUG -O2  ' '-DCMAKE_CXX_FLAGS_RELWITHDEBINFO=-O2 -g -DNDEBUG -ggdb  ' '-DCMAKE_CXX_STANDARD=17' '-DCMAKE_C_FLAGS= -Qunused-arguments -fPIC' '-DCMAKE_C_FLAGS_DEBUG=-g -Werror -O0 -ggdb -g1  -Wno-error' '-DCMAKE_C_FLAGS_MISIZEREL=-Os -DNDEBUG ' '-DCMAKE_C_FLAGS_RELEASE=-O3 -DNDEBUG -O2  ' '-DCMAKE_C_FLAGS_RELWITHDEBINFO=-O2 -g -DNDEBUG -ggdb  ' '-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=' '-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=' '-DCMAKE_INSTALL_LIBDIR=lib' '-DCMAKE_OSX_SYSROOT=/Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk' '-DCMAKE_VERBOSE_MAKEFILE=OFF' '-DCMAKE_C_COMPILER_LAUNCHER=/opt/homebrew/bin/ccache' '-DCMAKE_CXX_COMPILER_LAUNCHER=/opt/homebrew/bin/ccache' '-DCMAKE_INSTALL_PREFIX=/Users/runner/work/arrow/arrow/build/cpp/orc_ep-install' '-DSTOP_BUILD_ON_WARNING=OFF' '-DBUILD_LIBHDFSPP=OFF' '-DBUILD_JAVA=OFF' '-DBUILD_TOOLS=OFF' '-DBUILD_CPP_TESTS=OFF' '-DINSTALL_VENDORED_LIBS=OFF' '-DLZ4_HOME=/opt/homebrew/Cellar/lz4/1.9.4' '-DPROTOBUF_EXECUTABLE=/opt/homebrew/bin/protoc-27.0.0' '-DPROTOBUF_HOME=/opt/homebrew' '-DPROTOBUF_INCLUDE_DIR=/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '-DPROTOBUF_LIBRARY=/opt/homebrew/lib/libprotobuf.27.0.0.dylib' '-DPROTOC_LIBRARY=/opt/homebrew/lib/libprotoc.27.0.0.dylib' '-DSNAPPY_HOME=/opt/homebrew' '-DSNAPPY_LIBRARY=/opt/homebrew/lib/libsnappy.1.2.1.dylib' '-DLZ4_LIBRARY=/opt/homebrew/Cellar/lz4/1.9.4/lib/liblz4.dylib' '-DLZ4_STATIC_LIB=/opt/homebrew/Cellar/lz4/1.9.4/lib/liblz4.dylib' '-DLZ4_INCLUDE_DIR=/opt/homebrew/Cellar/lz4/1.9.4/include' '-DSNAPPY_INCLUDE_DIR=/opt/homebrew/include' '-DZSTD_HOME=/opt/homebrew' '-DZSTD_INCLUDE_DIR=/opt/homebrew/include' '-DZSTD_LIBRARY=/opt/homebrew/lib/libzstd.1.5.6.dylib' '-GNinja' '-S' '/Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep' '-B' '/Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-build'

  See also

    /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure-*.log

-- stdout output is:
-- The C compiler identification is AppleClang 15.0.0.15000040
-- The CXX compiler identification is AppleClang 15.0.0.15000040
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Build type: DEBUG
-- compiler AppleClang version 15.0.0.15000040
-- Disable the metrics collection
-- Performing Test ORC_CXX_HAS_CSTDINT
-- Performing Test ORC_CXX_HAS_CSTDINT - Success
-- Performing Test ORC_CXX_HAS_THREAD_LOCAL
-- Performing Test ORC_CXX_HAS_THREAD_LOCAL - Success
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- SNAPPY_HOME: /opt/homebrew
-- Found the Snappy header: /opt/homebrew/include/snappy.h
-- Found the Snappy library: /opt/homebrew/lib/libsnappy.1.2.1.dylib
-- ZSTD_HOME: /opt/homebrew
-- Found the zstd header: /opt/homebrew/include/zstd.h
-- Found the zstd library: /opt/homebrew/lib/libzstd.1.5.6.dylib
-- Found the zstd static library: /opt/homebrew/lib/libzstd.a
-- LZ4_HOME: /opt/homebrew/Cellar/lz4/1.9.4
-- Found the LZ4 header: /opt/homebrew/Cellar/lz4/1.9.4/include/lz4.h
-- Found the LZ4 library: /opt/homebrew/Cellar/lz4/1.9.4/lib/liblz4.dylib
-- Found the LZ4 static library: /opt/homebrew/Cellar/lz4/1.9.4/lib/liblz4.dylib
-- PROTOBUF_HOME: /opt/homebrew
-- ZLIB_HOME: 
-- Found the ZLIB header: /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-build/c++/libs/thirdparty/zlib_ep-install/include/zlib.h
-- Found the ZLIB library: /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-build/c++/libs/thirdparty/zlib_ep-install/lib/libz.a
-- Found the ZLIB static library: /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-build/c++/libs/thirdparty/zlib_ep-install/lib/libz.a
-- Configuring incomplete, errors occurred!

-- stderr output is:
CMake Warning:
  Ignoring extra path from command line:

   "/opt/homebrew/include"

CMake Warning:
  Ignoring extra path from command line:

   "/opt/homebrew/include"

CMake Error at cmake_modules/FindProtobuf.cmake:115 (message):
  Could not find Protobuf.  Looked in /opt/homebrew.
Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:417 (find_package)
  CMakeLists.txt:166 (INCLUDE)

CMake Error at /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure-DEBUG.cmake:47 (message):
  Stopping after outputting logs.

Component(s)

C++, Continuous Integration

raulcd commented 3 months ago

I've seen this happening also on non ARM64 jobs, example on this PR for macOS 12: Specifically this job: https://github.com/apache/arrow/actions/runs/9579913000/job/26413390696?pr=42207#step:8:891

drin commented 3 months ago

I have a customized version of the apache-arrow homebrew recipe that turns on substrait (apache-arrow-substrait)

I have been seeing this issue today and I couldn't figure out a workaround, so I'm going to try turning off ORC for myself (which the apache-arrow homebrew recipe currently has anyways).

Mostly commenting here since I arbitrarily linked to this issue (and that would probably seem fairly random without any context).

kou commented 3 months ago

@wgtmac I looked at this problem and opened an issue for this: https://issues.apache.org/jira/browse/ORC-1732 Could you take a look at it?

raulcd commented 3 months ago

It seems quite a lot of our verification jobs fail due to this issue :

CMake Error at cmake_modules/FindProtobuf.cmake:115 (message):
-- Found the ZLIB header: /var/folders/4z/th_0342d7sv9wl7kb3yfllq80000gn/T/arrow-HEAD.XXXXX.pGOTHd3O/cpp-build/orc_ep-prefix/src/orc_ep-build/c++/libs/thirdparty/zlib_ep-install/include/zlib.h
  Could not find Protobuf.  Looked in /usr/local.
-- Found the ZLIB library: /var/folders/4z/th_0342d7sv9wl7kb3yfllq80000gn/T/arrow-HEAD.XXXXX.pGOTHd3O/cpp-build/orc_ep-prefix/src/orc_ep-build/c++/libs/thirdparty/zlib_ep-install/lib/libz.a
Call Stack (most recent call first):
-- Found the ZLIB static library: /var/folders/4z/th_0342d7sv9wl7kb3yfllq80000gn/T/arrow-HEAD.XXXXX.pGOTHd3O/cpp-build/orc_ep-prefix/src/orc_ep-build/c++/libs/thirdparty/zlib_ep-install/lib/libz.a
  cmake_modules/ThirdpartyToolchain.cmake:417 (find_package)
-- Configuring incomplete, errors occurred!
  CMakeLists.txt:166 (INCLUDE)

verify-rc-source-cpp-macos-amd64 verify-rc-source-cpp-macos-arm64 verify-rc-source-integration-macos-amd64 verify-rc-source-integration-macos-arm64 verify-rc-source-python-macos-amd64 verify-rc-source-python-macos-arm64 verify-rc-source-ruby-macos-amd64 verify-rc-source-ruby-macos-arm64

raulcd commented 2 months ago

Issue resolved by pull request 43011 https://github.com/apache/arrow/pull/43011