getFileInfo call failed with FATAL_INVALID_RPC_HEADER (org.apache.hadoop.ipc.RpcServerException)

[xxh2000]

When I use this library to operate hdfs, the program is always error, the error is 2018/06/20 10:16:09 create /user/tnt/abc.txt: getFileInfo call failed with FATAL_INVALID_RPC_HEADER (org.apache.hadoop.ipc.RpcServerException)

the code is as follow

func main() {
        os.Setenv("HADOOP_NAMENODE","10.202.45.82:8020")
    os.Setenv("HADOOP_KRB_CONF","krb5.conf")
    os.Setenv("HADOOP_KEYTAB","xue.keytab")
    os.Setenv("HADOOP_SNAME","ds")
    client :=getClient()
    fw,err:=client.Create("/user/tnt/abc.txt")
    if err != nil{
        log.Fatal(err.Error())
    }
    fw.Write([]byte("hehehe"))
    fw.Close()
}
func getClient() *Client {
    nn := os.Getenv("HADOOP_NAMENODE")
    if nn == "" {
        log.Fatal("HADOOP_NAMENODE not set")
    }

    hadoopCfg := LoadHadoopConf("")
    krbClient := GetKrbClientIfRequired(hadoopCfg)

    options := ClientOptions{}

    options.Addresses = []string{nn}
    options.KerberosClient = krbClient
    options.ServicePrincipalName = GetServiceName()

    c, err := NewClient(options)
    if err != nil {
        log.Fatal(err)
    }

    return c
}

[Shastick]

Hello, and thanks for the feedback.

This error is frequent when the client uses the wrong RPC header, generally the non-Kerberos one.

We have a known issue where a problem in the krb5.conf will not trigger any warning (presumably because of a missing check somewhere.), yet will cause the hdfs client to not attempt a Kerberos connection: in our case it was caused by double quotes around the KDC domain name. (But your case might be entirely different).

This is most likely due to a nil being returned by GetKrbClientIfRequired(hadoopCfg) above: that is where I would start my investigation.

(I'm sorry I can't provide more insight, we currently don't have as much time as we would like for this)

[xxh2000]

When I debugged, I found that the sendInitialToken method in the sasl.go file returned an error. The error message was Networking_Error: TGS Exchange Error: issue sending TGS_REQ to KDC: KRB Error: (7) KDC_ERR_S_PRINCIPAL_UNKNOWN Server not found in Kerberos database whether or not it was related to the SPN. I didn't know how to fill in the SPN and randomly passed a string.

[Shastick]

The error you mention refers to the exchange with the ticket granting service: you need to pass the correct strings and be able to authenticate to the Kerberos server to proceed.

This seems more like an issue linked to your Kerberos setup and/or Kerberos access right.

[xxh2000]

I have solved the problem of Server not found in Kerberos database. It is really due to the problem of SPN format. But still get the error getFileInfo call failed with FATAL_INVALID_RPC_HEADER (org.apache.hadoop.ipc.RpcServerException), I found that debugging is due to an error in the doKerberosHandshake function, which readAuthReply function return an error, the error is wrong Token ID. Expected 0504, was 6030, I guess because the handshake did not succeed, so the back of the request will also be a problem.Do you have ideas now?

hava a look

[Shastick]

We only support a specific token type (identified by 0x0504, in this particular case. I can't remember the specific name from the RFC. If I recall correctly, this type of token is sent in clear text and only authenticated through a signature): Your server might use a different type if it is configured to use SASL in an encrypted mode (vs authenticated only).

I'm sorry if I can't be more helpful right now. The RFC's linked at the bottom of https://medium.com/sqooba/a-close-look-at-a-kerberos-handshake-authenticate-to-a-hadoop-namenode-via-hrpc-c56746ddb290 might help you: at least one of them should reference the 0504 constant, and this might give you more pointers regarding your situation.

Ultimately, if my hunch is correct and another type of token is used, we'll need to patch the lib to support it as well.