search

yahoo / storm-yarn

Storm-yarn enables Storm clusters to be deployed into machines managed by Hadoop YARN.
Other
417 stars 161 forks source link

Delegation Token Refreshes #65

Open ekohlwey opened 10 years ago

ekohlwey commented 10 years ago

According to https://issues.apache.org/jira/browse/YARN-941 - delegation tokens may be refreshed by simply restarting the application master. This makes intuitive sense.

I believe storm stores topology submission information in Zookeeper, so this could actually work quite well for storm on yarn in that the AM could inspect its delegation tokens and simply exit with a non-final error code every time the token expires. The topologies, etc would then be re-launched with the same processing and durability guarantees as normal storm (although, perhaps the restarts would be more noticeable).

revans2 commented 10 years ago

That is only part of YARN-941. The RM also needs an API to allow it to update the credentials it has cached in memory. That work is relatively small, but still needs to be done.

ekohlwey commented 10 years ago

do you know if thats being tracked somewhere?

revans2 commented 10 years ago

I filed YARN-941, and Steve came up with the great idea for an exit to trigger the cred refresh. I thought that teh need for the RM update was still listed as part of that JIRA. If it is confusing I am happy to hop on the JIRA and clarify the new scope of it. Also because I am always swamped with stuff if you want to take a crack at YARN-941 I would be happy to review/merge it in for you. Although most discussion around that should happen on the JIRA itself :)