search

grepplabs / kafka-proxy

Proxy connections to Kafka cluster. Connect through SOCKS Proxy, HTTP Proxy or to cluster running in Kubernetes.
Apache License 2.0
501 stars 87 forks source link

Multiple proxies with different client certificates #34

Closed mgusiew-guide closed 4 years ago

mgusiew-guide commented 4 years ago

Hi, I have configured Kafka with client certificate auth. I have multiple client private keys for client certificates. I can setup two kafka-proxies on different ports with different value for --tls-client-key-file parameter. The question is if such setup will work ? I noticed that in LoadBalancer example (https://gist.github.com/everesio/e6fae11ed69099ae3f867eddcd83db82) Kafka-proxy pods are exposed as singleton StatefulSets so just want to make sure that two instances which proxy different types of clients (based on client certificates) will not cause some conflicts/race conditions.

It is also interesting why StatefulSet is needed (the problem with StatefulSet is that when pod goes into unknown state manual intervention is required to decide if to replace it or not). Is it Kafka protocol that requires network identity to remain the same or statefulness of protocol calls or maybe something else ?

mgusiew-guide commented 4 years ago

FTR the clients have access to different topics, for example Client A has ACLs for topic A but not topic B and Client B has ACLs for topic B but not topic A

everesio commented 4 years ago

So far the proxy can load only one client certificate. This could be extended by providing a list of client certs to load https://github.com/grepplabs/kafka-proxy/blob/master/proxy/tls.go#L187

Each proxy works independently, so you can configure multiple proxy / proxy groups. The clients and proxies should be configured in that way that the traffic from different groups is not mixed.

There is no real need to use StatefulSet, you can use Deployment as well. Provided snippet sets up 1 proxy per broker. In this setup proxy must know mapping of other proxies (list of external-server-mappings ), so STS is a ways to provided consistent naming.

mgusiew-guide commented 4 years ago

Thanks for quick response @everesio !

Actually I don't need more than one client certificate per proxy group. I prefer each client to have separate proxy group, this is multi-tenant scenario so having each tenant proxy in different pod enables proxy resource limits (CPU, Memory, Open connections) on tenant level. Client connects to single proxy (tenant proxy) with proper client certificate (tenant certificate) so traffic is not mixed. So all good here.

Thanks for explaining the reason for StatefulSet, I agree it simplifies setup. Given my Kafka knowledge I am bit afraid about particular scenario: 1) Kafka with 3 brokers and 3 STS proxies, each proxy maps one broker as bootstrap and 2 others as external as in the example 2) Because of network problem one stateful set goes into unknown state In this scenario K8s will not automatically replace the pod (pod is not reachable) and will not add new one. From Kafka perspective all 3 brokers are healthy so Kafka will continue advertise healthy broker with unhealthy proxy, the other two proxies will also continue mapping to the unhealthy proxy. Manual intervention for unhealthy STS will be needed Feel free to clarify if I misunderstood anything

We use AWS so I am thinking about following alternative: 1) Setup EC2 with ENI for each proxy (ENI provides static private IP) 2) In case when EC2 is unhealthy launch new EC2 and reassign network interface to it This is just an idea, not sure how it works out (seems to me this is the approach that AWS MSK uses for replacing unhealthy nodes)

I also plan to try with K8s. However, as you mentioned without stateful set I will not have static network interface so it would be necessary to restart two other proxies in order to change external server mappings for replaced pod. Need to think more about it, maybe DNS or some K8s resources will address it.

Anyways things look promising. This project is really unique offering in Kafka proxies space right now :) Thank you for maintaining it

everesio commented 4 years ago

For more HA, you can always increase number replicas of each group (STS). Each replica from the same STS would represent the same broker node. Each K8S service would load balance ( for new connections) between the replicas of one STS.

mgusiew-guide commented 4 years ago

@everesio good to know that I can have more than one instance of proxy behind loadbalancer port. I wonder if this works in case of TLS and client certificates. We use AWS NLB with SSL passthrough so that client negotiates certificates with proxy and NLB (which is in the middle) just forwards the requests back and forth. AWS NLB does not have sticky sessions so may forward randomly, not sure if it is going to stick cert negotiation process with one proxy server and not fallback to another. We can test it though so thanks for the tip.

everesio commented 4 years ago

client will see only names of N load balancers (each LB can represent a Kafka broker). it is transparent if behind one LB there are M kafka proxies pods connecting to one kafka broker (together N*M proxy pods).
TLS cert are checked only during connection creation and connections are persistent.

mgusiew-guide commented 4 years ago

@everesio thank you for detailed explanation. This resolves my concerns.