Output Kafka is missing important parameter when using Kerberos

[akarneliuk]

Relevant telegraf.conf

[[outputs.kafka]]
  brokers = ["b-0.hl.kafka.svc:9092", "b-1.hl.kafka.svc:9092", "b-2.hl.kafka.svc:9092"]

  topic = "test-topic"

  client_id = "my-collector"

  version = "1.5.0"

  routing_tag = "source"

  ## Optional SASL:
  ## one of: OAUTHBEARER, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, GSSAPI
  ## (defaults to PLAIN)
  sasl_mechanism = "GSSAPI"

  ## used if sasl_mechanism is GSSAPI
  sasl_gssapi_service_name = "my_kafka_service"
  ## One of: KRB5_USER_AUTH and KRB5_KEYTAB_AUTH
  sasl_gssapi_auth_type = "KRB5_KEYTAB_AUTH"
  sasl_gssapi_kerberos_config_path = "/krb5/krb.conf"
  sasl_gssapi_realm = "realm"
  sasl_gssapi_key_tab_path = "/krb5/my.keytab"
  sasl_gssapi_disable_pafxfast = false

Logs from Telegraf

2024-01-01T12:07:03Z E! [agent] Failed to connect to [outputs.kafka], retrying in 15s, error was "kafka: invalid configuration (Net.SASL.GSSAPI.Username must not be empty when GSS-API mechanism is used)"

System info

Telegraf 1.29.1

Docker

No response

Steps to reproduce

1. Configure Kafka output in Telegraf using GSSAPI and Kerberos.
2. Authentication type is KRB5_KEYTAB_AUTH.
3. Try to connect to Kafka and produce data
4. It will fail.

Expected behavior

Telegraf connects to Kafka and start producing information collected from my relevant inputs

Actual behavior

Telegraf throw an error that there is no username provided. And there is no key in documentation, which suggests how to provide username. However, the underlying IBM library, which you use, requires this parameter: https://github.com/IBM/sarama/blob/main/kerberos_client.go#L42

You can see the very same error message in this test https://github.com/IBM/sarama/blob/main/config_test.go#L181, which relates to missing username.

Additional info

I suppose, the solution can be to expose all the possible keys (starting at least with username) from IBM/sarama, so that we can authenticate using Kerberos.

[powersj]

Hi,

And there is no key in documentation, which suggests how to provide username.

This is from the docs, right by the GSSAPI specific config options:

  ## Optional SASL Config
  # sasl_username = "kafka"
  # sasl_password = "secret"

I am not aware of a need to have a separate username/password config options, but continuing to use the existing config options seems correct. Maybe we need to make this more clear?

However, there are two items at play here:

1) In the upstream kafka library, sarama, it requires the username to be set. That error message you are currently getting is from the library itself, not telegraf. In your case, I do not think you need a username, but the upstream requires one. Is that correct? 2) The secret store changes destroy the sasl_username/sasl_password config options before they are used with gssapi config options. I have put up #14522 to resolve this part.

Could you try the artifacts in #14522, with a fake username for now, to see if that gets you further?

Thanks

[akarneliuk]

Hey @powersj ,

Thanks for looking into that one. I have built telegraf from your source and i am indeed a step further. There are some futher issues i face, but they are related now to gokrd5.

When do you think your patch could be incorporated in the official build?

Thanks, Anton

[powersj]

I will get this reviewed and in our bug fix release next week!