search

hortonworks-spark / spark-llap

Apache License 2.0
102 stars 68 forks source link

[SPARK-LLAP-198] Adds HiveServer2CredentialProvider into Spark LLAP #199

Closed HyukjinKwon closed 6 years ago

HyukjinKwon commented 6 years ago

What changes were proposed in this pull request?

This PR proposes to add Hive Server 2 credential provider.

How was this patch tested?

Unit tests added.

Closes #198

dongjoon-hyun commented 6 years ago

For this one, could you review and merge this, @jerryshao ?

EricWohlstadter commented 6 years ago

What is the expected usage of HiveServer2CredentialProvider? It's clear what it does, but not how to access it.

Do we instantiate it explicitly or does it get registered as described here:

"Spark supports integrating with other security-aware services through Java Services mechanism (see java.util.ServiceLoader). To do that, implementations of org.apache.spark.deploy.yarn.security.ServiceCredentialProvider should be available to Spark by listing their names in the corresponding file in the jar’s META-INF/services directory."

https://spark.apache.org/docs/2.3.0/running-on-yarn.html

jerryshao commented 6 years ago

Have we removed this code in Spark code base @dongjoon-hyun ?

HyukjinKwon commented 6 years ago

Not yet. Will do this soon by myself as soon as I am able. @dongjoon-hyun guided me before.

jerryshao commented 6 years ago

OK, sure. Please manually verify it on a secure cluster when this code in Spark is removed.

HyukjinKwon commented 6 years ago

@EricWohlstadter, just up to my knowledge, https://github.com/hortonworks-spark/spark-llap/pull/199#issuecomment-380916700 is true as written. Looks it's being loaded via YARNHadoopDelegationTokenManager by the java service mechanism (and then it sets user's credentials) from my reading. So, I added the test case with it.

BTW, Seems spark.security.credentials.hiveserver2.enabled has been false by default specifically for this one (spark.security.credentials.{service}.enabled's true by default).