Bad SASL negotiation status: 3 (GSS initiate failed)

[pksilen]

We experience an error when connecting to hive using Kerberos auth: Bad SASL negotiation status: 3 (GSS initiate failed)

Below is our code and before executing our code, we execute kinit:

kinit -kt {{ .Values.krb5.keytabFile }} {{ .Values.krb5.principal }};

    configuration.Username = cfg.HiveUsername
    configuration.Password = cfg.HivePassword
    configuration.Service = cfg.HiveService
    configuration.FetchSize = cfg.HiveFetchsize

    if cfg.HiveAuth == "KERBEROS" || cfg.HiveAuth == "kerberos" {
        configuration.TLSConfig = &tls.Config{
            InsecureSkipVerify: true,
        }
    }

    connection, errConn := gohive.Connect(cfg.HiveHost, cfg.HivePort, cfg.HiveAuth, configuration)
    if errConn != nil {
        return nil, fmt.Errorf("Could not connect to Hive. %v", errConn)
    }

    return &HiveClient{
        Configuration: configuration,
        Connection:    connection,
    }, nil

[pksilen]

Hi, Any chance to someone look at this problem? We earlier had the same code functioning ok. We are using gohive version 1.4.0. No in two environments we get this same issue. Can this be a configuration issue?

[beltran]

Hello, I missed this.

From the code you're using ssl and kerberos right? This should be working, you can see this test as an example of how to set this. My guess is that this is a configuration issue, seems like something is wrong with kerberos, it maybe the krb5.conf file. This is the one used for the tests. Also please check how the tests do kinit, which may solve your problem.

After you do kinit ..., what does klist display? Also useful would be the hive logs when you try to connect, there should be a stacktrace.

[beltran]

Hello, were you finally able to resolve this?

[pksilen]

It is not yet solved but is being investigated by our data lake expert. We found "Checksum failed" error in Hive logs:

ERROR org.apache.thrift.transport.TSaslTransport: [HiveServer2-Handler-Pool: Thread-47]: SASL negotiation failure2022-02-14 16:35:45,471 ERROR org.apache.thrift.transport.TSaslTransport: [HiveServer2-Handler-Pool: Thread-47]: SASL negotiation failurejavax.security.sasl.SaslException: GSS initiate failed at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199) ~[?:1.8.0_292] at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:537) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:652) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:649) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_292] at javax.security.auth.Subject.doAs(Subject.java:360) [?:1.8.0_292] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1855) [hadoop-common-3.0.0-cdh6.3.2.jar:?] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:649) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]Caused by: org.ietf.jgss.GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed) at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:858) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_292] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_292] ... 14 moreCaused by: sun.security.krb5.KrbCryptoException: Checksum failed at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:102) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94) ~[?:1.8.0_292] at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.(KrbApReq.java:149) ~[?:1.8.0_292] at sun.security.jgss.krb5.InitSecContextToken.(InitSecContextToken.java:140) ~[?:1.8.0_292] at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:831) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_292] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_292] ... 14 moreCaused by: java.security.GeneralSecurityException: Checksum failed at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256.decrypt(Aes256.java:76) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:100) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94) ~[?:1.8.0_292] at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.(KrbApReq.java:149) ~[?:1.8.0_292] at sun.security.jgss.krb5.InitSecContextToken.(InitSecContextToken.java:140) ~[?:1.8.0_292] at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:831) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_292] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_292] ... 14 more

[KeepFire8916]

Hello, did you finally resolve this ? I've got the same issue like this

[adslen]

The author has not handled the err here, and the actual errors are being concealed.

[beltran]

That is on purpose @adslen, the error may be set even when the context has been initialized successfully. But I'm happy to accept improvements if you can think of any.

[adslen]

We encountered a problem similar to this last time. We spent a considerable amount of time attempting to resolve it, only to discover that the 'err' wasn't handled here, and the actual error wasn't being thrown. After reviewing the code briefly, it seems that the situation you mentioned occurs only when the error is 'ErrContinueNeeded.' Perhaps we could handle this similarly to GORM's handling of 'gorm.ErrRecordNotFound,' where we throw the error and let the user decide whether to handle the exception. Like this:

[beltran]

Sorry you had to spend so much time, and thank you for your suggestion. Is this the fix you are proposing? If not I would appreciate if you could create the pull request.

[Azusain]

It is not yet solved but is being investigated by our data lake expert. We found "Checksum failed" error in Hive logs:

ERROR org.apache.thrift.transport.TSaslTransport: [HiveServer2-Handler-Pool: Thread-47]: SASL negotiation failure2022-02-14 16:35:45,471 ERROR org.apache.thrift.transport.TSaslTransport: [HiveServer2-Handler-Pool: Thread-47]: SASL negotiation failurejavax.security.sasl.SaslException: GSS initiate failed at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199) ~[?:1.8.0_292] at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:537) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:652) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:649) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_292] at javax.security.auth.Subject.doAs(Subject.java:360) [?:1.8.0_292] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1855) [hadoop-common-3.0.0-cdh6.3.2.jar:?] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:649) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]Caused by: org.ietf.jgss.GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed) at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:858) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_292] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_292] ... 14 moreCaused by: sun.security.krb5.KrbCryptoException: Checksum failed at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:102) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94) ~[?:1.8.0_292] at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.(KrbApReq.java:149) ~[?:1.8.0_292] at sun.security.jgss.krb5.InitSecContextToken.(InitSecContextToken.java:140) ~[?:1.8.0_292] at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:831) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_292] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_292] ... 14 moreCaused by: java.security.GeneralSecurityException: Checksum failed at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256.decrypt(Aes256.java:76) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:100) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94) ~[?:1.8.0_292] at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.(KrbApReq.java:149) ~[?:1.8.0_292] at sun.security.jgss.krb5.InitSecContextToken.(InitSecContextToken.java:140) ~[?:1.8.0_292] at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:831) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_292] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_292] ... 14 more

I've encountered the same issue and got the same log output from Hive server. How did you fix that?

[KeepFire8916]

Sorry，I couldn't solve this problem. I changed the firewall to control access permissions.

At 2024-05-30 16:28:44, "Azusain" @.***> wrote:

It is not yet solved but is being investigated by our data lake expert. We found "Checksum failed" error in Hive logs:

ERROR org.apache.thrift.transport.TSaslTransport: [HiveServer2-Handler-Pool: Thread-47]: SASL negotiation failure2022-02-14 16:35:45,471 ERROR org.apache.thrift.transport.TSaslTransport: [HiveServer2-Handler-Pool: Thread-47]: SASL negotiation failurejavax.security.sasl.SaslException: GSS initiate failed at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199) ~[?:1.8.0_292] at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:537) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:652) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:649) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_292] at javax.security.auth.Subject.doAs(Subject.java:360) [?:1.8.0_292] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1855) [hadoop-common-3.0.0-cdh6.3.2.jar:?] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:649) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]Caused by: org.ietf.jgss.GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed) at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:858) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_292] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_292] ... 14 moreCaused by: sun.security.krb5.KrbCryptoException: Checksum failed at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:102) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94) ~[?:1.8.0_292] at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.(KrbApReq.java:149) ~[?:1.8.0_292] at sun.security.jgss.krb5.InitSecContextToken.(InitSecContextToken.java:140) ~[?:1.8.0_292] at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:831) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_292] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_292] ... 14 moreCaused by: java.security.GeneralSecurityException: Checksum failed at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256.decrypt(Aes256.java:76) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:100) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94) ~[?:1.8.0_292] at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.(KrbApReq.java:149) ~[?:1.8.0_292] at sun.security.jgss.krb5.InitSecContextToken.(InitSecContextToken.java:140) ~[?:1.8.0_292] at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:831) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_292] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_292] ... 14 more

I've encountered the same issue and get the same log output from Hive server. How did you fix that?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[Azusain]

I've already solved this problem and it was my foolish mistake that caused it: I tried to export .keytab file by using 'ktadd' command and did't realize that it would make the KDC regenerate a new .keytab file, which appearently conflicted with the original one. So in the end I just copied the .keytab file from the keytab path set in the hive-site.xml to my client, and it worked fine...