search

apache / druid

Apache Druid: a high performance real-time analytics database.
https://druid.apache.org/
Apache License 2.0
13.52k stars 3.71k forks source link

AccessControlException problem in Hadoop index task (occurs with certain library) #17485

Open LEEAHRI opened 4 days ago

LEEAHRI commented 4 days ago

Affected Version

Druid 31.0.0

Description

Hello, I ran into a Kerberos authentication problem while trying to run a Hadoop index task. The problem seems to occur during the process of getting the MR job report. However, there is no direct issue with the task itself, and the task completes successfully, so the problem seems to be related to the authentication step rather than the execution of the task.

Below is the error stack trace (hadoop-client-api-3.3.6):

2024-11-18T15:11:03,789 WARN [task-runner-0-priority-0] org.apache.hadoop.ipc.Client - Exception encountered while connecting to the server {server:port}
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
    at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:179) ~[hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:392) ~[hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:561) ~[hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:347) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:783) ~[hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:779) ~[hadoop-client-api-3.3.6.jar:?]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_242]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_242]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) ~[hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:779) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:347) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1632) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.ipc.Client.call(Client.java:1457) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.ipc.Client.call(Client.java:1410) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139) [hadoop-client-api-3.3.6.jar:?]
    at com.sun.proxy.$Proxy263.getJobReport(Unknown Source) [?:?]
    at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:135) [hadoop-client-api-3.3.6.jar:?]
    at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_242]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_242]
    at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:326) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:430) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:872) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.mapreduce.Job$1.run(Job.java:333) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.mapreduce.Job$1.run(Job.java:330) [hadoop-client-api-3.3.6.jar:?]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_242]
    at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_242]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:330) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:614) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1737) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1699) [hadoop-client-api-3.3.6.jar:?]
    at org.apache.druid.indexer.DetermineHashedPartitionsJob.run(DetermineHashedPartitionsJob.java:136) [druid-indexing-hadoop-31.0.0-SNAPSHOT.jar:31.0.0-SNAPSHOT]

When I lowered the library version to hadoop-client:2.8.5 in the hadoopDependencyCoordinates, I could not reproduce the authentication issue and the task ran successfully with no errors.

  "hadoopDependencyCoordinates": ["org.apache.hadoop:hadoop-client:2.8.5"]

Is there any guidance you can provide on how to resolve the issue with the Kerberos authentication?

wujiandaod commented 2 days ago

I got the same issue with Druid-v27.0.0 and hadoop-client-api 3.3.6