search

TOSIT-IO / tdp-collection

Ansible collection to deploy the components of TDP
Apache License 2.0
21 stars 19 forks source link

HDFS broken when not using KMS #371

Closed Nuttymoon closed 2 years ago

Nuttymoon commented 2 years ago

When the ranger_kms service is disabled, HDFS NameNodes fail to start with;

ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.IOException: Could not instantiate KeyProvider for uri: NONE
    at org.apache.hadoop.util.KMSUtil.createKeyProviderFromUri(KMSUtil.java:74)
    at org.apache.hadoop.util.KMSUtil.createKeyProvider(KMSUtil.java:67)
    at org.apache.hadoop.hdfs.HdfsKMSUtil.createKeyProvider(HdfsKMSUtil.java:71)
    at org.apache.hadoop.hdfs.DFSUtil.createKeyProviderCryptoExtension(DFSUtil.java:1658)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:748)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:706)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)
2022-07-25 14:45:46,174 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.io.IOException: Could not instantiate KeyProvider for uri: NONE

This is due to this line: https://github.com/TOSIT-IO/tdp-collection/blob/9d3fbd86808279d9b97fdec60964a3f86e329fce/tdp_vars_defaults/hdfs/hdfs.yml#L56

mehdibn commented 2 years ago

@Nuttymoon please check this discussion : https://github.com/TOSIT-IO/tdp-collection/pull/365 you missed a commit in the master branch

leopaul36 commented 2 years ago

@mehdibn it' actually fixed in hadoop.yml but not in hdfs.yml. Which is why I had the issue in #365 and I thought I missed your fix!

Nuttymoon commented 2 years ago

Yes the problem comes from hdfs-site.xml