Upgrade to Debian 10, Java 11, Hadoop 3.3.1, build/run on ARM

[sebastian-nagel]

• upgrade to Debian 10, Java 11, Hadoop 3.3.1

• allow to build/run on ARM architecture (aarch64 / arm64)

• note: this PR includes #103, #104, #105

• tested both on x86_64 / amd64 and aarch64 / arm64

  • build (make)
  • launch cluster (docker-compose up)
  • UI (namenode, resourcemanager)
  • wordcount (make wordcount)

• status of native library support (here ARM):

  
  $> hadoop checknative
  2021-07-02 15:22:55,211 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
  2021-07-02 15:22:55,214 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
  2021-07-02 15:22:55,218 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
  2021-07-02 15:22:55,253 INFO nativeio.NativeIO: The native code was built without PMDK support.
  Native library checking:
  hadoop:  true /opt/hadoop-3.3.1/lib/native/libhadoop.so.1.0.0
  zlib:    true /lib/aarch64-linux-gnu/libz.so.1
  zstd  :  true /usr/lib/aarch64-linux-gnu/libzstd.so.1
  bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
  openssl: true /usr/lib/aarch64-linux-gnu/libcrypto.so
  ISA-L:   false libhadoop was built without ISA-L support
  PMDK:    false The native code was built without PMDK support.

$> hadoop version Hadoop 3.3.1 Source code repository https://github.com/apache/hadoop.git -r a3b9c37a397ad4188041dd80621bdeefc46885f2 Compiled by ubuntu on 2021-06-15T10:51Z Compiled with protoc 3.7.1 From source with checksum 88a4ddb2299aca054416d6b7f81ca55 This command was run using /opt/hadoop-3.3.1/share/hadoop/common/hadoop-common-3.3.1.jar

[jacobmstein]

I spent hours going through and fixing all these issues to try and run this on my M1 Mac. Finally found your PR, it fixes everything I missed. Thank you!

edit: Please note, this does not work on Raspberry Pi 4 with Ubuntu Server. I will get more details shortly, however it selects a non-existent build even though it's ARMv8. I had to modify the build selection logic.

[OneCricketeer]

Hadoop 3.3.3 officially supports Java 11 - https://hadoop.apache.org/docs/current/

[lemonGGGit]

I spent hours going through and fixing all these issues to try and run this on my M1 Mac. Finally found your PR, it fixes everything I missed. Thank you!

edit: Please note, this does not work on Raspberry Pi 4 with Ubuntu Server. I will get more details shortly, however it selects a non-existent build even though it's ARMv8. I had to modify the build selection logic.

@jacobmstein, Hello, Can you work it on 「Raspberry Pi 4 with Ubuntu Server」 now?