Ubuntu 18.04中源码配置安装KAE

[liusheng]

这里记录一下从源码安装和配置KAE来测试起对于Hadoop中性能的影响。KAE相关组件包含3个代码仓库：

• KAEdriver 整体KAE的内核驱动，安装过程实际上为加载几个驱动模块。
• KAE 基于openssl的engine机制，实现的一个openssl的插件，openssl通过配置后，可以通过KAE完成加速，最终也会使用KAEDriver驱动，利用硬件完成加速。
• KAEzip 对于zlib有侵入修改，然后zlib+KAEzip源码，最终生成一个修改后的zip，通过调用KAEDriver驱动，完成整个KAE加速流程。

0. 环境信息

• Ubuntu 18.04 server 64bit for ARM
• 32vCPUs 64GB Mem 500GB disk

1. 安装依赖

• 安装编译所需的依赖包

  sudo apt-get -q install -y build-essential autoconf automake libtool cmake pkg-config pciutils kmod
  sudo apt-get install linux-headers-`uname -r`

• 安装openssl

  sudo apt-get install -y openssl libssl-dev 

  并确认OpenSSL的版本>= 1.1.1a

  root@hadoop-kae:/opt/KAE_repos/KAE# dpkg -l|grep ssl
  ii  libflac8:arm64                        1.3.2-1                            arm64        Free Lossless Audio Codec - runtime C library
  ii  libssl-dev:arm64                      1.1.1-1ubuntu2.1~18.04.6           arm64        Secure Sockets Layer toolkit - development files
  ii  libssl1.0.0:arm64                     1.0.2n-1ubuntu5.3                  arm64        Secure Sockets Layer toolkit - shared libraries
  ii  libssl1.1:arm64                       1.1.1-1ubuntu2.1~18.04.6           arm64        Secure Sockets Layer toolkit - shared libraries
  ii  libzstd1:arm64                        1.3.3+dfsg-2ubuntu1.1              arm64        fast lossless compression algorithm
  ii  openssl                               1.1.1-1ubuntu2.1~18.04.5           arm64        Secure Sockets Layer toolkit - cryptographic utility
  ii  python3-openssl                       17.5.0-1ubuntu1                    all          Python 3 wrapper around the OpenSSL library

  2. 安装编译KAEdriver

• 编译 accelerator driver

  cd KAEdriver/kae_driver
  sudo make
  sudo make install
  sudo modprobe uacce
  sudo modprobe hisi_qm
  sudo modprobe hisi_sec2
  sudo modprobe hisi_hpre
  sudo modprobe hisi_zip
  sudo modprobe hisi_rde

  再确认一下内核模块是否成功加载

  sudo lsmod |grep uacce
  sudo lsmod |grep hisi_qm
  sudo lsmod |grep sgl
  sudo lsmod |grep hisi_sec2
  sudo lsmod |grep hisi_hpre
  sudo lsmod |grep hisi_zip

• 编译warpdrive

  cd KAEdriver/warpdrive
  sudo sh autogen.sh 
  sudo ./configure 
  sudo make 
  sudo make install

  确认

  hadoop@hadoop-aio:/opt/KAE_repos/KAEdriver/warpdrive$ ls -al /usr/local/lib |grep libwd
  -rw-r--r-- 1 root root 1231540 Sep  3 02:48 libwd.a
  -rwxr-xr-x 1 root root     917 Sep  3 02:48 libwd.la
  lrwxrwxrwx 1 root root      14 Sep  3 02:48 libwd.so -> libwd.so.1.3.8
  lrwxrwxrwx 1 root root      14 Sep  3 02:48 libwd.so.1 -> libwd.so.1.3.8
  -rwxr-xr-x 1 root root  669376 Sep  3 02:48 libwd.so.1.3.8

3. 编译安装KAE

这一个要保证OpenSSL和libssl-dev正确安装

• 编译KAE

  cd KAE
  sudo ./configure 
  sudo make  
  sudo make install

• 验证

  sudo openssl  genrsa -out test.key -engine kae 2048 
  sudo openssl  rsa -in test.key -pubout -out test_pub.key -engine kae 
  sudo openssl  rsautl -encrypt -in rsa_test -inkey test_pub.key -pubin -out rsa_test.en -engine kae 
  sudo openssl  rsautl -decrypt -in rsa_test.en -inkey test.key -out rsa_test.de -engine kae
  sudo openssl enc -sm4-cbc -a -in sm4_test -out sm4_test.en -pass pass:123456 -engine kae 
  sudo openssl enc -sm4-cbc -a -in sm4_test -out sm4_test.en -pass pass:123456 -p -engine kae 
  sudo openssl enc -sm4-cbc -d -a -in sm4_test.en -out sm4_test.de -pass pass:123456 -engine kae 
  sudo openssl sm3 -out sm3_out -engine kae sm3_test

  注意，在Ubuntu18.04下，上述验证会失败，是因为kae.so文件的路径不对，可以手动拷贝.so文件到对应的路径：

  sudo cp libkae.so.1.3.8 /usr/lib/aarch64-linux-gnu/engines-1.1/
  sudo ln -s /usr/lib/aarch64-linux-gnu/engines-1.1/libkae.so.1.3.8 /usr/lib/aarch64-linux-gnu/engines-1.1/kae.so
  sudo ln -s /usr/lib/aarch64-linux-gnu/engines-1.1/libkae.so.1.3.8 /usr/lib/aarch64-linux-gnu/engines-1.1/kae.so.0

  4. 安装zlib压缩库

  cd KAEzip
  sudo bash setup.sh install

5. 验证

可以通过lzbench结合“silesia.tar”来测试配置了KAE的zlib的性能

• 验证openssl的RSA性能

  openssl speed rsa2048
  openssl speed -engine kae rsa2048

  可以看出开启KAE以后，openssl RSA性能高了很多。

• 编译lzbench 需要在lzbench的Makefile中增加LDFLAGS += -lpthread -lz -L/usr/local/kaezip/lib 并且注释掉lzbench自带的zlib代码编译的语句

  
      ifeq ($(BUILD_STATIC),1)
              LDFLAGS += -lrt -static
              ifeq (1, $(shell [ "$(COMPILER)" = "gcc" ] && expr $(GCC_VERSION) \>= 80000))
                LDFLAGS += -lmvec
              endif
      endif
      LDFLAGS += -lpthread -lz -L/usr/local/kaezip/lib
  endif

...

ZLIB_FILES = zlib/adler32.o zlib/compress.o zlib/crc32.o zlib/deflate.o zlib/gzclose.o zlib/gzlib.o zlib/gzread.o

ZLIB_FILES += zlib/gzwrite.o zlib/infback.o zlib/inffast.o zlib/inflate.o zlib/inftrees.o zlib/trees.o

ZLIB_FILES += zlib/uncompr.o zlib/zutil.o

```shell
git clone https://github.com/inikep/lzbench
cd lzbench
sudo make

可以在作上述修改前后执行lzbench测试做前后对比

root@hadoop-kae:/opt/lzbench# ./lzbench -ezlib /opt/silesia.tar
lzbench 1.8 (64-bit Linux)   Assembled by P.Skibinski
Compressor name         Compress. Decompress. Compr. size  Ratio Filename
memcpy                   6307 MB/s  6266 MB/s   211947520 100.00 /opt/silesia.tar
zlib 1.2.11 -1             63 MB/s   216 MB/s    77259029  36.45 /opt/silesia.tar
zlib 1.2.11 -2             57 MB/s   222 MB/s    75002277  35.39 /opt/silesia.tar
zlib 1.2.11 -3             43 MB/s   228 MB/s    72967040  34.43 /opt/silesia.tar
zlib 1.2.11 -4             38 MB/s   225 MB/s    71002817  33.50 /opt/silesia.tar

root@hadoop-kae:/opt/lzbench# LD_LIBRARY_PATH=/usr/local/kaezip/lib/ ./lzbench -ezlib /opt/silesia.tar
lzbench 1.8 (64-bit Linux)   Assembled by P.Skibinski
Compressor name         Compress. Decompress. Compr. size  Ratio Filename
memcpy                   6026 MB/s  6509 MB/s   211947520 100.00 /opt/silesia.tar
zlib 1.2.11 -1           2283 MB/s  1473 MB/s    97072775  45.80 /opt/silesia.tar
zlib 1.2.11 -2           2281 MB/s  1469 MB/s    97076874  45.80 /opt/silesia.tar

在Hadoop集群中，可以在etc/hadoop/mapred-site.xml做如下配置来使能kaezip库

    <property>
      <name>mapreduce.map.env</name>
      <value>LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/kaezip/lib</value>
    </property>

6. Q&A

• 目前看来应该不能在容器中使用KAE，错误如下：

  root@hadoop-aio:/opt/lzbench# sudo openssl  genrsa -out test.key -engine kae 2048
  engine "kae" set.
  Generating RSA private key, 2048 bit long modulus (2 primes)
  get / path fail!
  cannot find available dev
  e is 65537 (0x010001)

[liusheng]

问题记录： 根据Hadoop的文档，当在yarn-site.xml中配置yarn.nodemanager.resource.detect-hardware-capabilities为true，则hadoop集群会自动探测系统的cpu和memory资源，但是在ARM集群上面尝试，探测的结果不正确。

    <property>
      <description>Enable auto-detection of node capabilities such as
      memory and CPU.
      </description>
      <name>yarn.nodemanager.resource.detect-hardware-capabilities</name>
      <value>true</value>
    </property>

实际根据nodemanager的启动日志，可以发现探测的结果为：

2020-09-17 16:32:11,913 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered with ResourceManager as hadoop-arm-kae-2:38733 with total resource of <memory:28629, vCores:1>

而实际上节点的配置为32vCPU+64G 内存。