search

prestodb / presto

The official home of the Presto distributed SQL query engine for big data
http://prestodb.io
Apache License 2.0
16.08k stars 5.39k forks source link

Select Hudi table failed in kerberos cluster #17178

Open dujl opened 2 years ago

dujl commented 2 years ago

When I select a hudi table in a kerberos cluster, it report GSS initiate failed.

Query 20220112_032437_00002_tmdm5 failed: Error checking path :hdfs://emr-master-1:8020/warehouse/tablespace/managed/hive/hudi_ctas_cow_pt_tbl/dt=2021-12-01/.hoodie_partition_metadata, under folder: hdfs://emr-master-1:8020/warehouse/tablespace/managed/hive/hudi_ctas_cow_pt_tbl/dt=2021-12-01
com.facebook.presto.spi.PrestoException: Error checking path :hdfs://emr-master-1:8020/warehouse/tablespace/managed/hive/hudi_ctas_cow_pt_tbl/dt=2021-12-01/.hoodie_partition_metadata, under folder: hdfs://emr-master-1:8020/warehouse/tablespace/managed/hive/hudi_ctas_cow_pt_tbl/dt=2021-12-01
    at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:128)
    at com.facebook.presto.hive.util.ResumableTasks.safeProcessTask(ResumableTasks.java:47)
    at com.facebook.presto.hive.util.ResumableTasks.access$000(ResumableTasks.java:20)
    at com.facebook.presto.hive.util.ResumableTasks$1.run(ResumableTasks.java:35)
    at com.facebook.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: Error checking path :hdfs://emr-master-1:8020/warehouse/tablespace/managed/hive/hudi_ctas_cow_pt_tbl/dt=2021-12-01/.hoodie_partition_metadata, under folder: hdfs://emr-master-1:8020/warehouse/tablespace/managed/hive/hudi_ctas_cow_pt_tbl/dt=2021-12-01
    at org.apache.hudi.hadoop.HoodieROTablePathFilter.accept(HoodieROTablePathFilter.java:188)
    at com.facebook.presto.hive.util.HiveFileIterator.lambda$getLocatedFileStatusRemoteIterator$0(HiveFileIterator.java:103)
    at com.google.common.collect.Iterators$5.computeNext(Iterators.java:639)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
    at com.facebook.presto.hive.util.HiveFileIterator.computeNext(HiveFileIterator.java:69)
    at com.facebook.presto.hive.util.HiveFileIterator.computeNext(HiveFileIterator.java:40)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
    at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1811)
    at java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:295)
    at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:207)
    at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:162)
    at java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:301)
    at java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
    at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:195)
    at com.facebook.presto.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:40)
    at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:121)
    ... 7 more
Caused by: org.apache.hudi.exception.HoodieException: Error checking Hoodie partition metadata for hdfs://emr-master-1:8020/warehouse/tablespace/managed/hive/hudi_ctas_cow_pt_tbl/dt=2021-12-01
    at org.apache.hudi.common.model.HoodiePartitionMetadata.hasPartitionMetadata(HoodiePartitionMetadata.java:143)
    at org.apache.hudi.hadoop.HoodieROTablePathFilter.accept(HoodieROTablePathFilter.java:139)
    ... 24 more
Caused by: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "emr-master-1/192.168.0.135"; destination host is: "emr-master-1":8020;
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
    at org.apache.hadoop.ipc.Client.call(Client.java:1480)
    at org.apache.hadoop.ipc.Client.call(Client.java:1413)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
    at com.sun.proxy.$Proxy265.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:776)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy266.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
    at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
    at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
    at org.apache.hadoop.fs.HadoopExtendedFileSystem.getFileStatus(HadoopExtendedFileSystem.java:384)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)
    at org.apache.hudi.common.model.HoodiePartitionMetadata.hasPartitionMetadata(HoodiePartitionMetadata.java:141)
    ... 25 more
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
    at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:688)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
    at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:651)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:738)
    at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529)
    at org.apache.hadoop.ipc.Client.call(Client.java:1452)
    ... 44 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
    at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:414)
    at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:561)
    at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:376)
    at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:730)
    at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:726)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:726)
    ... 47 more
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
    at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:162)
    at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
    at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:189)
    at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
    ... 56 more
houlingchen commented 2 years ago

I met the same question, we are using presto 0.265.1,what's the version of your presto?

houlingchen commented 2 years ago

I met the same question, we are using presto 0.256.1,what's the version of your presto?

BTW, kinit the principals that you set in the hive.properties, then start the presto, will fix this question.

dujl commented 2 years ago

I met the same question, we are using presto 0.265.1,what's the version of your presto?

0.260

houlingchen commented 2 years ago

https://github.com/prestodb/presto/issues/17703#issue-1218406603

pratyakshsharma commented 1 year ago

@dujl were you able to resolve this issue?