Open tomfans opened 1 year ago
Hello @tomfans, Thanks for finding the time to report the issue! We really appreciate the community's efforts to improve Apache Kyuubi.
The differences may come from Spark client/cluster mode.
spark-sql
and spark-shell
only support client mode, while Kyuubi supports both client and cluster mode, there are some differences in the kerberized cluster.
By default, Kyuubi uses --proxy-user
instead of --principal
and --keytab
, so on
HadoopDelegationTokenProvider
) and distributed delegation tokens during spark-submit
phase.Something extended to this issue, Kyuubi implemented DSv2 based Hive connector(a.k.a. KSHC).
And in #4560
... make Kyuubi Spark Hive Connector(KSHC) support kerberized-HMS in cluster mode w/o keytab(which is the typical use case in Kyuubi) by implementing a
HadoopDelegationTokenProvider
.
There are some notable tricks
spark-sql
has some inconsistent behaviors on HiveClient initialization, which makes inconsistent behavior when you using spark-sql
for testing. Jar-based Spark application, spark-shell
and beeline
+ Kyuubi work well.hive.metastore.token.signature
for different HMS to distinguish the delegation tokens, otherwise the latter will overwrite the former. In #4560, we use the metastore uri as the signature for KSHC catalog if hive.metastore.token.signature
is not set explicitly. So technically, to allow Iceberg to use different kerberized-HMS, you can register an additional KSHC catalog, and make sure they use the same metastore uri and signature, thus they can share the delegation token.
Code of Conduct
Search before asking
Describe the bug
the case is after configuring iceberg metastore which is different from hive metastore(means i have two different hive metastores), when connecting iceberg catalog. metastore can't be connected cause delegation token expire errors.
why i said this is bug, since when i use original spark-sql command, it works fine.
here examples:
but kyuubi-1.7 failed.
Affects Version(s)
1.7
Kyuubi Server Log Output
No response
Kyuubi Engine Log Output
No response
Kyuubi Server Configurations
No response
Kyuubi Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?