[Meta] Supporting Kerberos in Elastic Agent Kafka modules

[nimarezainia]

Currently there are gaps in agent/beats support for kerberos in various kafka implementations. This meta issue is to gather all the specific development efforts that need to be implemented in order to provide a cohesive offering

### Tasks
- [ ] Update the Kafka client currently in use by beats/agent so that we have access to the latest
- [ ] Ensure filebeat Kafka input correctly authenticates against Active Directory using Kerberos config
- [ ] Ensure *beats output to Kafka authenticates against Active Directory using Kerberos config, and make this feature GA
- [ ] Modify the Fleet UI for Kafka output to include a Kerberos configuration
- [ ] (optional) Support kerberos authentication on the Metricbeat kafka module```

Related issus:

• [ ] https://github.com/elastic/beats/issues/33334
• [ ] https://github.com/elastic/elastic-agent-shipper/issues/168
• [ ] https://github.com/elastic/enhancements/issues/21809

cc: @cmacknz

[cmacknz]

The first step here will be getting a test environment with both Kafka and Kerberos using Active Directory.

We can then test if we can fix this by upgrading our fork of https://github.com/elastic/sarama which implements our Kafka client. This is quite far (~700 commits) behind upstream now. There is an upstream Kerberos test we could use as a starting point https://github.com/IBM/sarama/blob/main/kerberos_client_test.go. This appears to have been added in https://github.com/IBM/sarama/pull/2457.

We can simulataneously evaluate if it is worth continuing to maintain our own fork of Sarama, or even evaluate alternative client packages, although that will present challenges with configuration backwards compatibility. https://github.com/twmb/franz-go is one alternative that claims Kerberos support.

[ycombinator]

Moving this to the beats repo as the majority of the work is over there.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[nimarezainia]

The first step here will be getting a test environment with both Kafka and Kerberos using Active Directory.

We can then test if we can fix this by upgrading our fork of https://github.com/elastic/sarama which implements our Kafka client. This is quite far (~700 commits) behind upstream now. There is an upstream Kerberos test we could use as a starting point https://github.com/IBM/sarama/blob/main/kerberos_client_test.go. This appears to have been added in IBM/sarama#2457.

We can simulataneously evaluate if it is worth continuing to maintain our own fork of Sarama, or even evaluate alternative client packages, although that will present challenges with configuration backwards compatibility. https://github.com/twmb/franz-go is one alternative that claims Kerberos support.

kafka exporter in otel seems to have support for kerberos although there's no way of telling whether it works with AD. How would one tell which implementation of the kafka client is being used there?

Thinking ahead, if we were to utilise otel for this feature I see the following priorities:

1. Ensure kafka exporter has all the features we need for the support provided today (whether upstreamed or in our dist)
2. Change our Fleet UI (beats based) kafka implementation to interface with the otel shipper instead
3. Modify the standalone config to be otel based (which would be a breaking change unless we do some translation between beats to otel config)

Which ever way we choose to go, building a testing environment for this first is a must.

[cmacknz]

It is the same as ours, github.com/IBM/sarama https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/5427c934ebcad5dfb77bca48e44b063349b6e4f4/internal/kafka/authentication.go#L12

[nimarezainia]

It is the same as ours, github.com/IBM/sarama https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/5427c934ebcad5dfb77bca48e44b063349b6e4f4/internal/kafka/authentication.go#L12

trying to make a decision on whether this can wait until otel shipper is available and we switch over. Even if we assume that the otel kafka exporter has an updated fork of the client and that there are no issues with it working with AD, we won't be able to use this for the filebeat Kafka input. Perhaps that will be the caveat we live with.