Support Azure Data Lake Storage Gen2 (ADLS)

[siegfriedweber]

Support Azure Data Lake Storage Gen2 (ADLS)

Required changes in the cluster spec

ADLS replaces HDFS. A custom "HDFS" ConfigMap can be used instead of the one provided by an HDFS cluster.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: adls
data:
  core-site.xml: |-
    <?xml version="1.0"?>
    <configuration>
      <property>
        <name>fs.defaultFS</name>
        <value>abfs://<container-name>@<storage-account>.dfs.core.windows.net/</value>
      </property>
      <property>
        <name>fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net</name>
        <value>SAS</value>
      </property>
      <property>
        <name>fs.azure.sas.fixed.token.<storage-account>.dfs.core.windows.net</name>
        <value>${env.SAS_TOKEN}</value>
      </property>
      <!-- further properties, e.g.: -->
      <property>
        <name>hadoop.rpc.protection</name>
        <value>privacy</value>
      </property>
      <property>
        <name>hadoop.security.authentication</name>
        <value>kerberos</value>
      </property>
    </configuration>
  hdfs-site.xml: |-
    <?xml version="1.0"?>
    <configuration>
    </configuration>

Required changes in the image

• The fixed SAS token provider should be added to the hadoop-azure library. HADOOP-18516 provides this and can be back-ported.
• The Azure storage JARs must be copied from the Hadoop image.

These changes were already made as a test in https://github.com/stackabletech/docker-images/commit/7314d4f194603408f00b555c2e98ff956c212d85.

Tasks

• [x] stackabletech/docker-images#852
• [x] stackabletech/docker-images#853
• [x] stackabletech/hbase-operator#560