ARM64 系统中编译安装Hadoop

安装Java和Maven

和编译Hive类似，源码编译Hadoop，需要先安装Java和Maven。

安装Java

#apt-get install default-jre -y
#apt-get install default-jdk -y
apt-get install openjdk-8-jdk -y
java_home=$(dirname $(dirname $(update-alternatives --list javac)))
echo "export JAVA_HOME=${java_home}" >> ~/.profile
echo "export PATH=${java_home}/bin:$PATH" >> ~/.profile
source ~/.profile

安装Maven3

# install maven-3
wget "https://archive.apache.org/dist/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz"
tar xzvf apache-maven-3.6.1-bin.tar.gz
new_path=$(realpath apache-maven-3.6.1/bin/):$PATH
grep -q 'export PATH=' ~/.profile && sed -i "s|^export PATH=.*$|export PATH=${new_path}|" ~/.profile || echo "export PATH=${new_path}" >> ~/.profile
source ~/.profile
echo $PATH
mvn --version

安装protoc

直接编译Hadoop的话，会报错，由于其依赖的protoc-2.5.0版本在Maven的仓库没有ARM64版本的，需要手动编译并打上[补丁](protobuf-2.5.0-arm64.patch https://www.dropbox.com/s/713wql5cw9dfxhx/protobuf-2.5.0-arm64.patch?dl=0)：

apt-get install -y build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev libsasl2-dev
wget https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.bz2
# the original patch has bug when building hadoop native libary
# wget -O protobuf-2.5.0-arm64.patch https://www.dropbox.com/s/713wql5cw9dfxhx/protobuf-2.5.0-arm64.patch?dl=0
wget -O protobuf-2.5.0-arm64.patch https://git.io/JeObp
tar xvfj protobuf-2.5.0.tar.bz2
cd protobuf-2.5.0
patch -p1 < ../protobuf-2.5.0-arm64.patch
./autogen.sh
./configure --prefix=/usr
make
make check
make install
protoc --version

由于编译过程比较耗时，为了方便起见，我编译好了一份，后面可以直接下载试用(可以在上述命令中指定--profix指定一个单独的目录，用来编译输出的文件)： protoc-2.5.0-aarch64.tar.gz

protobuf-2.5.0-aarch64.tar.gz

curl -Lo /opt/protobuf-2.5.0-aarch64.tar.gz  https://git.io/JeOpb
mkdir -p /opt/protobuf-2.5.0-aarch64 && tar zxf /opt/protobuf-2.5.0-aarch64.tar.gz -C /opt/protobuf-2.5.0-aarch64/
echo /opt/protobuf-2.5.0-aarch64/lib/ > /etc/ld.so.conf.d/protobuf-2.5.0-aarch64.conf
ldconfig
export PATH=/opt/protobuf-2.5.0-aarch64/bin:$PATH
protoc --version

Netty的问题： netty-all-4.1.27-linux-aarch64.jar.tar.gz

liusheng / liusheng.github.io

ARM64 系统中编译安装Hadoop #12

安装Java和Maven

安装protoc