Kafka REST Proxy with Load balancer for consuming messages

[Capconv]

Hi, I'm running multiple instances of Kafka Rest Proxy and wanted to know how to setup the Load Balancer for running multiple instances on different servers.

My use case is to consume messages from Kafka Broker, just wanted to know that is there any configuration in Rest Proxy where we get the hostname in the base_uri as response when the consumer is created while using non sticky load balancer.

Example:

Request for creating consumer :

POST http://loadbalancer.kafkaproxy.example.com:8081/consumers/testgroup with the body as below

{ "name": "my_consumer", "format": "protobuf", "auto.offset.reset": "earliest", "auto.commit.enable": "false" }

Response :

{ "instance_id": "my_consumer", "base_uri": "http://loadbalancer.kafkaproxy.example.com:8081/consumers/testgroup/instances/my_consumer" }

But the expected response should be :

{ "instance_id": "my_consumer", "base_uri": "http://hostname.kafkaproxy.example.com:8081/consumers/testgroup/instances/my_consumer" }

Referring the documentation below where it was mentioned that "then uses the returned hostname on subsequent requests"

Generally, you need a sticky load balancer session because consumers are stateful. Therefore, out-of-the-box REST Proxy requires that the load balancer is sticky; that is, each consumer instance should always be routed to the same REST Proxy instance.

However, you can use a stateless connection if your consumer sends only the initial request to the load balancer, and then uses the returned hostname on subsequent requests.

[Capconv]

@dimitarndimitrov Can you please help with the above issue.

[dimitarndimitrov]

Hey @Capconv, have you configured advertised.listeners on your Kafka REST instances?

The base URI can be configured with the advertised.listeners (or the deprecated host.name + port) configs. If neither of these is configured, the base URI uses the hostname provided from the URI info of the request hitting the corresponding endpoint (see here).

[Capconv]

Hi @dimitarndimitrov,

Thanks for the response I figured out after a long time that I had forgotten to add the host header in the request Also, wanted to know is there any documentation or reference related to sticky session load balancer configuration with Kafka Rest Proxy and the best practices related to the same.

Thanks a lot.

[echooymxq]

@Capconv Do you have a solution to this problem?