search

paypal / NNAnalytics

NameNodeAnalytics is a self-help utility for scouting and maintaining the namespace of an HDFS instance.
Apache License 2.0
110 stars 71 forks source link

Document set-up for 3.x Hadoop based installation #205

Open pjeli opened 5 years ago

pjeli commented 5 years ago

After having just experimented with a 3.x based cluster I have found that it may be required to run with nna.support.bootstrap.overrides=false if you are running security (Kerberos) and that you should provide a custom dfs.ha.namenode.id to satisfy the bootstrap process.

Ideally these challenges should go away when NNA becomes a simple ServicePlugin but until then we should document this so others can have an easier time with the current RPM installation.

pjeli commented 1 month ago

If you are running into the following issues "Could not determine own NameNode HA ID" or "dfs.namenode.lazypersist.file.scrub.interval.sec must be non-zero" or "Class org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer not found" do the following on the NNA hdfs-site.xml:

  1. Set block secret manager enabled to false. NNA does not need block tokens anyway as it does NOT talk to DataNodes.
  2. Set "dfs.namenode.lazypersist.file.scrub.interval.sec" to -1 to satisfy bootstrap. Again, we are not going to be doing any writes / modifications via NNA. It is purely reading it's own metadata.
  3. If you have "dfs.namenode.inode.attributes.provider.class" set, remove it. NNA does not provide Ranger support or require Ranger look-ups.