Support consumer, producer, listener and template specific Kafka Admin properties

[hemantmaersk]

The current implementation of KafkaProperties in Spring Boot allows for the configuration of different producer and consumer instances, enabling the reading from different cluster and writing to altogether different Kafka cluster using various user IDs and passwords. However, a limitation exists wherein only one admin configuration can be set for both producers and consumers. This constraint becomes problematic in various scenarios requiring cluster id. Observability, is one such instance.

To address this limitation, developers should be empowered to configure different Admin instances for producers and consumers independently. This modification becomes crucial for scenarios where distinct cluster IDs are required for enhanced observability.

To summarise,The current functionality do empower developers is to read from different cluster by configuring spring.kafka.consumer and can write to altogether different cluster using spring.kafka.producer however there is only one admin instance that can be configured for both via spring.kafka.admin

Basically it would be great if we could have something like below

@ConfigurationProperties(prefix = "spring.kafka")
public class KafkaProperties {

    /**
     * Comma-delimited list of host:port pairs to use for establishing the initial
     * connections to the Kafka cluster. Applies to all components unless overridden.
     */
    private List<String> bootstrapServers = new ArrayList<>(Collections.singletonList("localhost:9092"));

    /**
     * ID to pass to the server when making requests. Used for server-side logging.
     */
    private String clientId;

    /**
     * Additional properties, common to producers and consumers, used to configure the
     * client.
     */
    private final Map<String, String> properties = new HashMap<>();

    /**
     * Configuration for Kafka consumers.
     */
    private final Consumer consumer = new Consumer();

    /**
     * Configuration for Kafka producers.
     */
    private final Producer producer = new Producer();

    /**
     * Configuration for Kafka Admin.
     */
    private final Admin producerAdmin = new Admin();

    /**
     * Configuration for Kafka Admin.
     */
    private final Admin consumerAdmin = new Admin();
}

[philwebb]

Currently the admin property triggers the creation of a single KafkaAdmin bean. If we were to create more than one admin bean we'd need to find a way to associate the correct admin bean with either a producer or consumer. It's not clear that this is something we can currently do with spring-kafka.

@artembilan Does this suggestion make sense? Would we need updates to Spring Kafka to support it?

[artembilan]

I see the point of the request. However it supposed to be slightly different.

The KafkaTemplate use KafkaAdmin to resolve just only clusterId(), which has nothing to do with the ApplicationContext. So, we could just have KafkaProperties.Template.admin of Admin type. Then in the KafkaAutoConfiguration.kafkaTemplate() we would perform setKafkaAdmin() based on that admin property. And this without any bean creation. If KafkaTemplate.setKafkaAdmin() is provided, no any ApplicationContext consultation happens.

Same logic for the KafkaProperties.Listener.kafkaAdmin & ConcurrentKafkaListenerContainerFactoryConfigurer.configureContainer(). And we don't need a dedicated bean again.