kimoonkim
commented
7 years ago Perhaps @ifilonenko @foxish would be most interested. There is one sore point in the token refresh server prototype. I used the akka library for message-driven actor thread model, which I think simplified the code a lot. (akka is the standard actor library for Scala)
But this is not a good thing to do for Spark as app framework, which moved away from akka for dependency management reason. See details at http://apache-spark-developers-list.1001551.n3.nabble.com/Why-did-spark-switch-from-AKKA-to-net-td21522.html.
More specifically, many user applications that link to Spark also linked to Akka as a library (e.g. say you want to write a service that receives requests from Akka and runs them on Spark). In that case, you'd have two conflicting versions of the Akka library in the same JVM.
But it's a fine thing to do for the token refresh server, which is an independent service app and has little to do with Spark. So I think it's best for the token refresh server to live outside the Spark repo. Or, if it's too much work, keep it as an independent project with its own pom.xml with no dependencies to/from Spark.
Thoughts?
liyinan926
ifilonenko
@ifilonenko @liyinan926
Implements a standalone server, named hadoop token refresh server, that extends the lifetimes of HDFS tokens stored in K8s Secrets. This is part of the secure HDFS support.
HDFS tokens expire after every 24 hours or some configured time period. You can renew them until max time 7 days. After that, you need to obtain brand new tokens.
This refresh server takes care of both short term token renewal and obtaining new tokens.
The server runs in a k8s pod. There is a
Dockerfilefor building docker images and a k8sdeploymentyaml file for launching the pod.This code exists in a separate maven project with its own pom.xml file. You can build a tarball using:
For details, see README.md
The refresh server has three parallel components:
Token Refresh Service: An actor-based service that takes discovered secrets as input and schedules tasks to renew tokens in the secrets. Other components below send commands to this service.Secret Scanner: A periodic thread that scans and discovers secrets that have a well-known label key value pair.refresh-hadoop-tokens: "yes"Secret Watcher: An event watcher that discovers new or deleted secrets that have the said label.Although its role overlaps with the secret watcher, the secret scanner is needed for bootstrapping and recovery:
Each renew task is in charge of one or more tokens stored in a K8s secret. When it runs, the task does:
Tested manually so far. (I'll add unit tests in this PR. I plan to add integration tests later when both the other secure HDFS PR and this PR are merged in)
Here's the log that shows how a brand new token is obtained:
After the above run, the K8s secret now has two data items, with the latest item containing the new token: