search

grepplabs / kafka-proxy

Proxy connections to Kafka cluster. Connect through SOCKS Proxy, HTTP Proxy or to cluster running in Kubernetes.
Apache License 2.0
501 stars 87 forks source link

Question: Kubernetes: External IP, Service? #30

Closed brunodomenici closed 5 years ago

brunodomenici commented 5 years ago

Hi,

Is there any recommendation for deploy Kafka-proxy in Kubernetes and using a Service type LoadBalancing? I did deploy it but I have to bind to and external IP and I don't have it on Pod. I can't map to my LoadBalance hostname/ip:

 --bootstrap-server-mapping=original-bootstrap:9092,external-ip-hostname:9092",

Any ideas? Thanks a lot!

everesio commented 5 years ago

Hi, There is an optional third parameter in bootstrap-server-mapping and you can use it as external name which will be advertised back to client. --bootstrap-server-mapping "kafka-0.example.com:9092,0.0.0.0:32401,kafka-0.grepplabs.com:9092" There must be a separate endpoint for each Kafka node. i.e. you need N * TCP Loadbalancers or one TCP with N different tcp ports / listeners.

I tested and used configuration with N Loadbalancers on GCP.

everesio commented 5 years ago

I forgot to mention that you must also map other nodes with external-server-mapping on each proxy instance

- '--bootstrap-server-mapping=kafka-{{ $i }}-0:9093,0.0.0.0:32400,kafka-gateway-{{ $i }}-0.{{ $.dnszone }}:32400'
{{- range $j := until (int $.brokers) }}
- '--external-server-mapping=kafka-{{ $j }}-0:9093,kafka-gateway-{{ $j }}-0.{{ $.dnszone }}:32400'
{{- end }}

Check out the https://gist.github.com/everesio/e6fae11ed69099ae3f867eddcd83db82 it should give you the idea.

brunodomenici commented 5 years ago

Thanks a lot @everesio. Thats solved my problem.