When using HDFS as journal to support the HA mode, I pass in a principal and keytab via the alluxio.master.keytab.file and alluxio.master.principal config. However the principal and keytab is not used when authenticating against HDFS, causing Alluxio to fail to format the journal on HDFS, and thus Alluxio master cannot start in HA mode.
To Reproduce
Configure HDFS to require Kerberos auth. Use HDFS as journal and start in HA mode.
Expected behavior
When using HDFS as journal, Alluxio should use the provided principal and keytab to perform Kerberos auth to HDFS.
Urgency
When using HDFS as journal, and the HDFS requires Kerberos auth, Alluxio master cannot start in HA mode.
Are you planning to fix it
No
Additional context
Error messages
2022-08-25 02:43:53,852 ERROR Format - Failed to format
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "phx2-qh5.prod.uber.internal/10.80.78.36"; destination host is: "hadoopplatinumnamenode01-phx2":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:805)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511)
at org.apache.hadoop.ipc.Client.call(Client.java:1453)
at org.apache.hadoop.ipc.Client.call(Client.java:1363)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:796)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1649)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1529)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1526)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1526)
at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1641)
at alluxio.underfs.hdfs.HdfsUnderFileSystem.isDirectory(HdfsUnderFileSystem.java:497)
at alluxio.underfs.UnderFileSystemWithLogging$30.call(UnderFileSystemWithLogging.java:687)
at alluxio.underfs.UnderFileSystemWithLogging$30.call(UnderFileSystemWithLogging.java:684)
at alluxio.underfs.UnderFileSystemWithLogging.call(UnderFileSystemWithLogging.java:1237)
at alluxio.underfs.UnderFileSystemWithLogging.isDirectory(UnderFileSystemWithLogging.java:684)
at alluxio.master.journal.ufs.UfsJournal.format(UfsJournal.java:449)
at alluxio.master.journal.ufs.UfsJournalSystem.format(UfsJournalSystem.java:245)
at alluxio.cli.Format.format(Format.java:121)
at alluxio.cli.Format.main(Format.java:95)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:760)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:723)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:817)
at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:412)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1568)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
... 30 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:407)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:618)
at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:412)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:804)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:800)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:799)
... 33 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:162)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:189)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 42 more
Alluxio Version:
2.9.0-SNAPSHOT up to this commit on 8/4/2022 https://github.com/Alluxio/alluxio/commit/fdcba75c7e65bf812c2ba07b7c968ef3c9550213
Describe the bug
When using HDFS as journal to support the HA mode, I pass in a principal and keytab via the
alluxio.master.keytab.file
andalluxio.master.principal
config. However the principal and keytab is not used when authenticating against HDFS, causing Alluxio to fail to format the journal on HDFS, and thus Alluxio master cannot start in HA mode.To Reproduce
Configure HDFS to require Kerberos auth. Use HDFS as journal and start in HA mode.
Expected behavior
When using HDFS as journal, Alluxio should use the provided principal and keytab to perform Kerberos auth to HDFS.
Urgency
When using HDFS as journal, and the HDFS requires Kerberos auth, Alluxio master cannot start in HA mode.
Are you planning to fix it
No
Additional context
Error messages