search

apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
https://gluten.apache.org/
Apache License 2.0
1.01k stars 363 forks source link

I can't build gluten on an Intel MacOS machine #5491

Closed NEUpanning closed 1 week ago

NEUpanning commented 3 weeks ago

Problem description

I ran ./dev/buildbundle-veloxbe.sh and it failed.

System information

Velox System Info v0.0.2 Commit: 1b219aaa3d14e34f794c8b4369aa11603169460c CMake Version: 3.29.2 System: Darwin-23.2.0 Arch: x86_64 C++ Compiler: C++ Compiler Version: 15.0.0.15000100 C Compiler: /Library/Developer/CommandLineTools/usr/bin/cc C Compiler Version: 15.0.0.15000100 CMake Prefix Path:

CMake log

++ cd /Users/panning/VSProjects/glutenNvelox_CE/incubator-gluten/dev/../cpp
++ rm -rf build
++ mkdir build
++ cd build
++ cmake -DBUILD_VELOX_BACKEND=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_BENCHMARKS=OFF -DBUILD_JEMALLOC=OFF -DENABLE_HBM=OFF -DENABLE_QAT=OFF -DENABLE_IAA=OFF -DENABLE_GCS=OFF -DENABLE_S3=OFF -DENABLE_HDFS=OFF ..
-- Building using CMake version: 3.29.2
-- The C compiler identification is AppleClang 15.0.0.15000100
-- The CXX compiler identification is AppleClang 15.0.0.15000100
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Set VELOX_HOME to /Users/panning/VSProjects/glutenNvelox_CE/incubator-gluten/ep/build-velox/build/velox_ep
-- Add definition NDEBUG
Unsupported compiler ID: AppleClang
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance
-- Found JNI: /Library/Java/JavaVirtualMachines/jdk1.8.0_301.jdk/Contents/Home/include  found components: AWT JVM
-- Found glog: /usr/local/lib/libglog.dylib
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
-- Found Boost: /usr/local/lib/cmake/Boost-1.84.0/BoostConfig.cmake (found version "1.84.0")
-- Set Substrait Proto Directory in /Users/panning/VSProjects/glutenNvelox_CE/incubator-gluten/gluten-core/src/main/resources/substrait/proto
-- Set Gluten Proto Directory in /Users/panning/VSProjects/glutenNvelox_CE/incubator-gluten/gluten-core/src/main/resources/io/glutenproject/proto
-- Found Arrow library: /Users/panning/VSProjects/glutenNvelox_CE/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/third_party/arrow_ep/install/lib/libarrow.1400.dylib
-- Found Arrow library: /Users/panning/VSProjects/glutenNvelox_CE/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/third_party/arrow_ep/install/lib/libparquet.1400.dylib
-- Could NOT find Protobuf (missing: Protobuf_INCLUDE_DIR)
-- Use existing ProtoBuf libraries: /usr/local/lib/libprotobuf.a
-- Use existing Jemalloc libraries
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
-- Found Boost: /usr/local/lib/cmake/Boost-1.84.0/BoostConfig.cmake (found suitable version "1.84.0", minimum required is "1.51.0") found components: context filesystem program_options regex system thread
-- Found folly: /usr/local
-- Found Arrow library: /Users/panning/VSProjects/glutenNvelox_CE/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/third_party/arrow_ep/install/lib/libarrow.1400.dylib
-- Found Arrow library: /Users/panning/VSProjects/glutenNvelox_CE/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/third_party/arrow_ep/install/lib/libparquet.1400.dylib
-- Could NOT find re2 (missing: re2_DIR)
-- RE2 Library Can Be Found in /usr/local/lib/libre2.a
-- Found Thrift: /usr/local/lib/libthrift.dylib (found version "0.20.0")
-- Configuring done (3.1s)
CMake Error at core/CMakeLists.txt:296 (add_dependencies):
  The dependency target "protobuf::libprotobuf" of target "jni_proto" does
  not exist.

-- Generating done (0.2s)
CMake Generate step failed.  Build files cannot be regenerated correctly.
xumingming commented 3 weeks ago

The direct cause is: Unsupported compiler ID: AppleClang, I think AppleClang is already supported, try pull the code.

NEUpanning commented 3 weeks ago

@xumingming Thanks for your reply. I pulled the latest code and the build failed again. Here is cmake log

+ echo ENABLE_S3=OFF,ENABLE_GCS=OFF,ENABLE_HDFS=OFF,ENABLE_ABFS=OFF,BUILD_TYPE=Release,VELOX_HOME=/Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/ep/build-velox/src/../build/velox_ep,ENABLE_BENCHMARK=OFF,ENABLE_TESTS=OFF,BUILD_TEST_UTILS=OFF,COMPILE_ARROW_JAVA=OFF,OTHER_ARGUMENTS=,COMMIT_HASH=3a997a5934059e77dc6c0ee13e48946f5cbad015
++ build_gluten_cpp
++ echo 'Start to Gluten CPP'
Start to Gluten CPP
++ cd /Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/dev/../cpp
++ rm -rf build
++ mkdir build
++ cd build
++ cmake -DBUILD_VELOX_BACKEND=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_BENCHMARKS=OFF -DBUILD_JEMALLOC=OFF -DENABLE_HBM=OFF -DENABLE_QAT=OFF -DENABLE_IAA=OFF -DENABLE_GCS=OFF -DENABLE_S3=OFF -DENABLE_HDFS=OFF -DENABLE_ABFS=OFF ..
-- Building using CMake version: 3.29.2
-- The C compiler identification is AppleClang 15.0.0.15000309
-- The CXX compiler identification is AppleClang 15.0.0.15000309
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Set VELOX_HOME to /Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/ep/build-velox/build/velox_ep
-- Add definition NDEBUG
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found JNI: /Library/Java/JavaVirtualMachines/jdk1.8.0_301.jdk/Contents/Home/include  found components: AWT JVM
-- Found glog: /usr/local/lib/libglog.dylib
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
-- Found Boost: /usr/local/lib/cmake/Boost-1.84.0/BoostConfig.cmake (found version "1.84.0")
-- Set Substrait Proto Directory in /Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/gluten-core/src/main/resources/substrait/proto
-- Set Gluten Proto Directory in /Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/gluten-core/src/main/resources/org/apache/gluten/proto
-- Found Arrow library: /Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/third_party/arrow_ep/install/lib/libarrow.1500.dylib
-- Found Arrow library: /Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/third_party/arrow_ep/install/lib/libparquet.1500.dylib
-- Could NOT find Protobuf (missing: Protobuf_INCLUDE_DIR)
-- Use existing ProtoBuf libraries: /usr/local/lib/libprotobuf.a
-- Use existing Jemalloc libraries
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
-- Found Boost: /usr/local/lib/cmake/Boost-1.84.0/BoostConfig.cmake (found suitable version "1.84.0", minimum required is "1.51.0") found components: context filesystem program_options regex system thread
-- Found folly: /usr/local
-- Found Arrow library: /Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/third_party/arrow_ep/install/lib/libarrow.1500.dylib
-- Found Arrow library: /Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/third_party/arrow_ep/install/lib/libparquet.1500.dylib
-- RE2 Library Can Be Found in re2::re2
-- Found Thrift: /usr/local/lib/libthrift.dylib (found version "0.20.0")
-- Configuring done (4.1s)
CMake Error at core/CMakeLists.txt:295 (add_dependencies):
  The dependency target "protobuf::libprotobuf" of target "jni_proto" does
  not exist.

-- Generating done (0.3s)
CMake Generate step failed.  Build files cannot be regenerated correctly.

It seems that make can't find Protobuf but i already installed it. Here is brew list command output.

(python_envs) ➜  incubator-gluten git:(main) brew list protobuf
/usr/local/Cellar/protobuf/26.1/bin/protoc
/usr/local/Cellar/protobuf/26.1/bin/protoc-26.1.0
/usr/local/Cellar/protobuf/26.1/include/google/ (239 files)
/usr/local/Cellar/protobuf/26.1/include/java/core/src/main/resources/google/protobuf/java_features.proto
/usr/local/Cellar/protobuf/26.1/include/upb_generator/mangle.h
/usr/local/Cellar/protobuf/26.1/include/ (2 files)
/usr/local/Cellar/protobuf/26.1/lib/libprotobuf-lite.26.1.0.dylib
/usr/local/Cellar/protobuf/26.1/lib/libprotobuf.26.1.0.dylib
/usr/local/Cellar/protobuf/26.1/lib/libprotoc.26.1.0.dylib
/usr/local/Cellar/protobuf/26.1/lib/cmake/ (10 files)
/usr/local/Cellar/protobuf/26.1/lib/pkgconfig/ (3 files)
/usr/local/Cellar/protobuf/26.1/lib/ (5 other files)
/usr/local/Cellar/protobuf/26.1/share/emacs/site-lisp/protobuf/protobuf-mode.el
/usr/local/Cellar/protobuf/26.1/share/protobuf/ (25 files)
/usr/local/Cellar/protobuf/26.1/share/vim/vimfiles/syntax/proto.vim
xumingming commented 3 weeks ago

Two suggestions here:

  1. Velox needs protobuf@21, install the correct version of protobuf.
  2. Gluten is complex, try build Velox successfully first, then build gluten.
NEUpanning commented 3 weeks ago

@xumingming Thanks for your suggestion.

  1. I already installed protobuf@21. Here is the output of brew list protobuf 
    
➜  velox git:(note) ✗ brew search protobuf
==> Formulae
protobuf ✔                       protobuf-c ✔                     protobuf@21 ✔                    protobuf@3 ✔                     swift-protobuf

==> Casks protopie

2. Through the cmake log i can see velox is built successfully. I tried to build gluten cpp separately and got same error as above. Here is cmake log

➜ build git:(main) cmake -DBUILD_VELOX_BACKEND=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_BENCHMARKS=OFF -DBUILD_JEMALLOC=OFF -DENABLE_HBM=OFF -DENABLE_QAT=OFF -DENABLE_IAA=OFF -DENABLE_GCS=OFF -DENABLE_S3=OFF -DENABLE_HDFS=OFF -DENABLE_ABFS=OFF .. -- Building using CMake version: 3.29.2 -- Set VELOX_HOME to /Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/ep/build-velox/build/velox_ep -- Add definition NDEBUG -- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) -- Found Boost: /usr/local/lib/cmake/Boost-1.84.0/BoostConfig.cmake (found version "1.84.0") -- Set Substrait Proto Directory in /Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/gluten-core/src/main/resources/substrait/proto -- Set Gluten Proto Directory in /Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/gluten-core/src/main/resources/org/apache/gluten/proto -- Found Arrow library: /Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/third_party/arrow_ep/install/lib/libarrow.1500.dylib -- Found Arrow library: /Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/third_party/arrow_ep/install/lib/libparquet.1500.dylib -- Could NOT find Protobuf (missing: Protobuf_INCLUDE_DIR) -- Use existing ProtoBuf libraries: /usr/local/lib/libprotobuf.a -- Use existing Jemalloc libraries -- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) -- Found Boost: /usr/local/lib/cmake/Boost-1.84.0/BoostConfig.cmake (found suitable version "1.84.0", minimum required is "1.51.0") found components: context filesystem program_options regex system thread -- Found folly: /usr/local -- Found Arrow library: /Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/third_party/arrow_ep/install/lib/libarrow.1500.dylib -- Found Arrow library: /Users/panning/VSProjects/glutenNvelox_CE/gluten/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/third_party/arrow_ep/install/lib/libparquet.1500.dylib -- RE2 Library Can Be Found in re2::re2 -- Configuring done (0.5s) CMake Error at core/CMakeLists.txt:295 (add_dependencies): The dependency target "protobuf::libprotobuf" of target "jni_proto" does not exist.

acvictor commented 2 weeks ago

Can you try installing PkgConfig as well?