Hive 在ARM64平台上的编译

[liusheng]

1. 安装JDK和Maven3

• 安装Java

  apt-get install default-jre -y
  apt-get install default-jdk -y
  java_home=$(dirname $(dirname $(update-alternatives --list javac)))
  echo "export JAVA_HOME=${java_home}" >> ~/.profile
  echo "export PATH=${java_home}/bin:$PATH" >> ~/.profile
  source ~/.profile

• 安装Maven3

  # install maven-3
  wget "https://archive.apache.org/dist/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz"
  tar xzvf apache-maven-3.6.1-bin.tar.gz
  new_path=$(realpath apache-maven-3.6.1/bin/):$PATH
  grep -q 'export PATH=' ~/.profile && sed -i "s|^export PATH=.*$|export PATH=${new_path}|" ~/.profile || echo "export PATH=${new_path}" >> ~/.profile
  source ~/.profile
  echo $PATH
  mvn --version

  2. 编译Hive

  mvn -B -e clean package -Pdist -DskipTests

  直接编译会报下面的错误：

  [ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.5.1.1:run (default) on project hive-standalone-metastore-common: Error resolving artifact: com.google.protobuf:protoc:2.5.0: Could not find artifact com.google.protobuf:protoc:exe:linux-aarch_64:2.6.1-build3 in central (https://repo.maven.apache.org/maven2)
  [ERROR] 

  这是由于Hive依赖于protoc-2.5.0的库在Maven的central repo, 没有ARM64版本的，有两种方式来规避该问题，一种是修改依赖库的版本，另一种是手动编译。

规避方式一：

直接编译，也会报错，需要打上补丁，脚本如下：

# install dependencies for protobuf build
apt-get install autoconf automake libtool -y
# build protobuf-2.5.0 for ARM64 arch
pushd /usr/local/src/
wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
# Fix protobuf-2.5.0 building on ARM64
wget -O protobuf-2.5.0-arm64.patch https://www.dropbox.com/s/713wql5cw9dfxhx/protobuf-2.5.0-arm64.patch?dl=0
tar xfz protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
patch -p1 < ../protobuf-2.5.0-arm64.patch
apt-get install build-essential
apt-get install automake libtool autoconf
./autogen.sh
./configure --prefix=/usr
make
make install
popd
ldconfig
protoc --version

然后再修改Hive中的protoc:2.5.0的引用位置，指向本地编译好的protoc二进制文件：

sed -i 's|<protocArtifact>com.google.protobuf:protoc:2.5.0</protocArtifact>|<protocCommand>/usr/bin/protoc</protocCommand>|g' `grep "protoc:2.5.0" -rl ./`
git diff

规避方式二：

修改Hive中com.google.protobuf:protoc:2.5.0的引用为com.github.os72:protoc:2.6.1-build3

sed -i 's|com.google.protobuf:protoc:2.5.0|com.github.os72:protoc:2.6.1-build3|g' `grep "protoc:2.5.0" -rl ./`
git diff

该包可以在这里搜到。

重新编译Hive

再次执行mvn -B -e clean package -Pdist -DskipTests，编译成功：

[INFO] Hive Storage API 2.7.0-SNAPSHOT .................... SUCCESS [ 35.316 s]
[INFO] Hive 4.0.0-SNAPSHOT ................................ SUCCESS [  3.588 s]
[INFO] Hive Classifications 4.0.0-SNAPSHOT ................ SUCCESS [  2.908 s]
[INFO] Hive Shims Common 4.0.0-SNAPSHOT ................... SUCCESS [ 13.182 s]
[INFO] Hive Shims 0.23 4.0.0-SNAPSHOT ..................... SUCCESS [ 21.488 s]
[INFO] Hive Shims Scheduler 4.0.0-SNAPSHOT ................ SUCCESS [  8.195 s]
[INFO] Hive Shims 4.0.0-SNAPSHOT .......................... SUCCESS [  7.796 s]
[INFO] Hive Standalone Metastore 4.0.0-SNAPSHOT ........... SUCCESS [  7.144 s]
[INFO] Hive Standalone Metastore Common Code 4.0.0-SNAPSHOT SUCCESS [01:31 min]
[INFO] Hive Common 4.0.0-SNAPSHOT ......................... SUCCESS [ 26.369 s]
[INFO] Hive Service RPC 4.0.0-SNAPSHOT .................... SUCCESS [ 10.166 s]
[INFO] Hive Serde 4.0.0-SNAPSHOT .......................... SUCCESS [ 22.236 s]
[INFO] Hive Metastore 4.0.0-SNAPSHOT ...................... SUCCESS [ 15.598 s]
[INFO] Hive Vector-Code-Gen Utilities 4.0.0-SNAPSHOT ...... SUCCESS [  1.390 s]
[INFO] Hive Llap Common 4.0.0-SNAPSHOT .................... SUCCESS [ 12.416 s]
[INFO] Hive Llap Client 4.0.0-SNAPSHOT .................... SUCCESS [  7.765 s]
[INFO] Hive Llap Tez 4.0.0-SNAPSHOT ....................... SUCCESS [  9.947 s]
[INFO] Hive Spark Remote Client 4.0.0-SNAPSHOT ............ SUCCESS [ 31.137 s]
[INFO] Hive Metastore Server 4.0.0-SNAPSHOT ............... SUCCESS [01:22 min]
[INFO] Hive Query Language 4.0.0-SNAPSHOT ................. SUCCESS [04:10 min]
[INFO] Hive Llap Server 4.0.0-SNAPSHOT .................... SUCCESS [ 34.095 s]
[INFO] Hive Service 4.0.0-SNAPSHOT ........................ SUCCESS [ 39.474 s]
[INFO] Hive Accumulo Handler 4.0.0-SNAPSHOT ............... SUCCESS [ 32.670 s]
[INFO] Hive JDBC 4.0.0-SNAPSHOT ........................... SUCCESS [01:23 min]
[INFO] Hive Beeline 4.0.0-SNAPSHOT ........................ SUCCESS [ 23.243 s]
[INFO] Hive CLI 4.0.0-SNAPSHOT ............................ SUCCESS [ 13.128 s]
[INFO] Hive Contrib 4.0.0-SNAPSHOT ........................ SUCCESS [  7.813 s]
[INFO] Hive Druid Handler 4.0.0-SNAPSHOT .................. SUCCESS [01:31 min]
[INFO] Hive HBase Handler 4.0.0-SNAPSHOT .................. SUCCESS [ 21.637 s]
[INFO] Hive JDBC Handler 4.0.0-SNAPSHOT ................... SUCCESS [  8.978 s]
[INFO] Hive HCatalog 4.0.0-SNAPSHOT ....................... SUCCESS [  5.312 s]
[INFO] Hive HCatalog Core 4.0.0-SNAPSHOT .................. SUCCESS [ 20.405 s]
[INFO] Hive HCatalog Pig Adapter 4.0.0-SNAPSHOT ........... SUCCESS [ 14.177 s]
[INFO] Hive HCatalog Server Extensions 4.0.0-SNAPSHOT ..... SUCCESS [ 13.020 s]
[INFO] Hive HCatalog Webhcat Java Client 4.0.0-SNAPSHOT ... SUCCESS [ 13.702 s]
[INFO] Hive HCatalog Webhcat 4.0.0-SNAPSHOT ............... SUCCESS [ 51.169 s]
[INFO] Hive HCatalog Streaming 4.0.0-SNAPSHOT ............. SUCCESS [ 15.080 s]
[INFO] Hive HPL/SQL 4.0.0-SNAPSHOT ........................ SUCCESS [ 18.665 s]
[INFO] Hive Streaming 4.0.0-SNAPSHOT ...................... SUCCESS [ 10.964 s]
[INFO] Hive Llap External Client 4.0.0-SNAPSHOT ........... SUCCESS [ 10.416 s]
[INFO] Hive Shims Aggregator 4.0.0-SNAPSHOT ............... SUCCESS [  0.345 s]
[INFO] Hive Kryo Registrator 4.0.0-SNAPSHOT ............... SUCCESS [ 11.091 s]
[INFO] Hive TestUtils 4.0.0-SNAPSHOT ...................... SUCCESS [  0.563 s]
[INFO] Hive Kafka Storage Handler 4.0.0-SNAPSHOT .......... SUCCESS [ 29.490 s]
[INFO] Hive Packaging 4.0.0-SNAPSHOT ...................... SUCCESS [03:31 min]
[INFO] Hive Metastore Tools 4.0.0-SNAPSHOT ................ SUCCESS [  0.037 s]
[INFO] Hive Metastore Tools common libraries 4.0.0-SNAPSHOT SUCCESS [ 53.578 s]
[INFO] Hive metastore benchmarks 4.0.0-SNAPSHOT ........... SUCCESS [03:06 min]
[INFO] Hive Upgrade Acid 4.0.0-SNAPSHOT ................... SUCCESS [  0.043 s]
[INFO] Hive Pre Upgrade Acid 4.0.0-SNAPSHOT ............... SUCCESS [ 21.233 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  28:30 min
[INFO] Finished at: 2019-06-21T07:17:05Z

下面为Patch文件： hive-protoc-patch.tar.gz