Simplify load balancing of kafka-rest with haproxy

[edoardocomar]

Given a cluster of kafka-rest servers, it would be desirable to allow haproxy to

• spread consumer instance creation POST /consumers/(string: group_name)
• persist consumer instance usage (GET, DELETE,POST offset) to the only server instance that holds the created consumer
• spread all other calls across the cluster

At the moment, the simplest approach would be to have a cookie set by the load balancer. Perhaps thsi could become insteadt part of the kafka-rest API by having it set by the server and documented.

see also http://blog.haproxy.com/2012/03/29/load-balancing-affinity-persistence-sticky-sessions-what-you-need-to-know/

[kjvalencik]

Creating a consumer instance returns the hostname (base_uri) of the instance. Are your instances not individually and predictably addressable? You could set HAProxy to hash on source ip.

http://docs.confluent.io/1.0/kafka-rest/docs/api.html#post--consumers-(string-group_name)

[edoardocomar]

Hi Kjvalencik,

• There could be many clients behind one source ip.
• The rest instances may not be directly addressable (eg only the haproxies have public IPs)

[kjvalencik]

That makes sense. The cookie thing would definitely work. A couple more options:

• Unique port for each Kafka REST instance
• Layer 7 routing based on directory or sub-domain.

I've only done layer 3 proxying in HAProxy, but here would be the equivalent in nginx:

upstream kafka_rest_upstream {
    server 1.kafka-rest.local:8082;
    server 2.kafka-rest.local:8082;
    server 3.kafka-rest.local:8082;
}

server {
    server_name kafka-rest.local;
    listen 8082;

    location /(\d+)/ {
        proxy_pass http://$1.kafka-rest.local:8082/;
    }

    location / {
        proxy_pass http://kafka_rest_upstream;
    }
}

Subscribing would then return "base_uri" : "1.kafka-rest.local:8082" and you would re-write that as kafka-rest.local:8082/1/