Strimzi client support for Go, Ruby, Python

[klalafaryan]

Hello!

We would like to use strimzi kafka with Oauth support. To do this we need to have a support for multiple languages like Java, Go, Ruby, Python.

I can see that the Java client is available, but what about other languages ?

Can we use vanilla kafka-client for other languages or not ? If not, what other options do we have ?

Thanks, Konstantin

[scholzj]

TBH, I'm not sure how is this done in other languages. What you really need is for the clients to support the SASL OAUTHBEARER mechanism. If that is supported, you need to get the access token from your OAuth server and use it.

@mstruk Did you ever checked how does it look in librdkafka and so on?

[mstruk]

@scholzj No I haven't checked. It's like you say, as long as a client library support SASL OAUTHBEARER there is a foundation we can work with and extend. We'll have to check and create some examples how to achieve that with different language runtimes.

[klalafaryan]

@scholzj @mstruk Thanks for your reply.

Just one more thing to clarify. Is it possible to use apache kafka (vanilla) java client for Oauth with strimzi server ? (without using strimzi-kafka client)

Also I can see that strimzi kafka-oauth-client is coupled with keycloak. Can you please give some more insights ? Thanks.

[scholzj]

The Java client provides support for the SASL OAUTHBEARER mechanism. This project adds the callbacks to talk with the OAuth servers, obtain the tokens (+ in the broker validate the token) etc. So out of the box the Java client has some preparation, but does not work out of the box. I have no idea what the state is for the other programming language clients.

Keycloak is one of the OAuth servers we support and we test with. We use the Keycloak client library to tak with the OAuth servers - it is really just an implementation of the OAuth protocol / interface. But for authentication which is part of the actual OAuth, it works also with other OAuth servers, not only with Keycloak. The Authorizer which is part of this repository currently works only with Keycloak (since authorization is not really so well standadizes as OAuth authentication, we had to start somewhere).

[klalafaryan]

@scholzj Thanks for the clarification!

One alternative is to try the confluent client. I am not sure if they have out of box oauth support in the other clients.

Do you have some plans to have strimzi-client for Go (and other languages) ? Thanks.

[scholzj]

TBH we didn't thought abotu it yet. So it is probably fair to say it is not on our plan. We would need to find time for it and figure out how feasible it is (or isn't). The Java clients were written from beginning to make this plugable. Not sure whether for example Go has the same or how is it expected to work there. We would need to have a look. So it is probably not something to wait for if you need a solution right now.

Of course, any PRs and contributions are always welcomed ;-).

[uwburn]

Hello, i found this very interesting project for auth in Kafka, that accomplish exactly what i was looking for. For me interoperability with other environment is critical, so i share what i found out.

Personally, i'm using mostly Node.JS, and i'm submitting a PR to kafkajs lib so that it can properly follow the SASL/OAUTHBEARER mechanism.

Once a client is able to do SASL/OAUTHBEARER there's nothing special to add, it will just work. You just need to follow the client spec in providing the token. I tested it with Keycloak and had no problems with it.

Python client, C librdkafka and GO confluent-kafka-go also support SASL/OAUTHBEARER. Unfortunately some clients (even ones based on librdkafka) still can't leverage it.

[HackRanger]

@uwburn I am getting the token from keycloak serve by making a HTTP post and passing the token to Kafka golang client by calling p.SetOAuthBearerToken(authtoken) . But still I get invalid token error. Is it the correct way of fetching the token and passing it to kafka?

[2020-03-30 18:01:09,623] DEBUG {"status":"invalid_token"} (org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslServer)

type KeycloakTokenResp struct {
    AccessToken      string `json:"access_token"`
    ExpiresIn        int    `json:"expires_in"`
    RefreshExpiresIn int    `json:"refresh_expires_in"`
    RefreshToken     string `json:"refresh_token"`
    TokenType        string `json:"token_type"`
    NotBeforePolicy  int    `json:"not-before-policy"`
    SessionState     string `json:"session_state"`
    Scope            string `json:"scope"`
}

func getToken() *KeycloakTokenResp{
    username := "subbus.joshi@gmail.com"
    password := "welcome123"
    client_id := "kafka"
    client_secret := "52e55800-3f28-4332-b252-8dac9b7ae40f"

    var val url.Values
    val = make(map[string][]string, 0)
    val.Add("username",username)
    val.Add("password",password)
    val.Add("client_id",client_id)
    val.Add("client_secret",client_secret)
    val.Add("grant_type","token-exchange")

    resp, err := http.PostForm("http://localhost:8080/auth/realms/scania/protocol/openid-connect/token",val)
    defer resp.Body.Close()
    body, err := ioutil.ReadAll(resp.Body)
    if err != nil{
        log.Println(err.Error())
    } else{
        log.Println(string(body))

        var tokenResp KeycloakTokenResp

        json.Unmarshal(body,&tokenResp)
        return &tokenResp
    }
    return nil
}

[uwburn]

@HackRanger, i don't know GO, the only things i can comment are:

• I use grant_type client_credentials to obtain the token (don't know if it's relevant here)
• I only send the access_token string contained in the JSON response (that is what Kafka expects)

The rest should be up to the client library, if it does the job correctly or if it does expect something different.

While adding the support to the Node.js client library i used Wireshark to track the content of what the Java client was producing and what i was producing... it's a bit cumbersome to do, but it does definitely help.