search

paypal / NNAnalytics

NameNodeAnalytics is a self-help utility for scouting and maintaining the namespace of an HDFS instance.
Apache License 2.0
110 stars 71 forks source link

Provide easier restart functionality #275

Closed pjeli closed 5 years ago

pjeli commented 5 years ago

Today if you "stop" a long-running NNA process, there will be several ramifications:

(1) You will likely need to run /fetchNamespace endpoint to get a new, fresh, more up-to-date FsImage. (2) You will then, afterwards, need to run /reloadNamespace to load in the new FsImage and play the edit logs from the JournalNodes in order to catch up fully.

It should be possible to provide a start-up routine for NNA where it will automatically do all of these functions for you on start-up so that you don't need to them manually each restart.

One possibility for now is to simply provide a /fetchAndReloadNamespace API function that is just a merge of the two calls above. This would be significant enough. However I would like to hopefully also just make this call upon bootstrap.

Need to think of what the best way to do this would be. As obviously for some restarts we would like to NOT do the fetch and reload again and again.

pjeli commented 5 years ago

Should this new functionality be automatically called upon restart? I feel that the answer is yes. Maybe it can be controlled via configuration? Need to think further.

pjeli commented 5 years ago

This is now committed. In version 1.6.6 of NNA you can figure to auto-fetch on restart by setting "nna.bootstrap.auto.fetch.namespace" to "true" in your application.properties file.