search

vivekrsharma / kstream-experiments

This is to play around with kstreams on top of Kafka
Apache License 2.0
0 stars 0 forks source link

Compute end to end latency of the setup w/ {producer, processor and consumer} #8

Closed vivekrsharma closed 2 years ago

vivekrsharma commented 2 years ago

Let's try the following -

  1. Setup Kafka cluster in - say, aws - us-west-2, and expose it over either VPC peering or Private Link
  2. Setup ec2 instance in same region and connect to this kafka using VPC peering or PrivateLink
  3. Ensure that the packet routing between ec2 and kafka never leaves the geography boundaries
  4. Check end to end latency for this setup
vivekrsharma commented 2 years ago

This has been a great experimentation. We were able to track the latency from 1500 ms down to almost 50ms by having a more finer grain control over the routing of packets from producer/processor/consumer to the Kafka cluster.

Also note that each of the three components make at least a handful of round trips to the Kafka cluster. More prominent one is processor with all the intermediate Kafka topic creation etc.

What we had found in our initial experimentation is that the packets would leave geography even though both the client and server would be in same geography.

We then used VPC-Peering ang Private Link offerings from Confluent Cloud to have a more fine grain control on the routing of traffic.

We were able to get the latency down to 50 ms.

Kudos to https://docs.confluent.io/cloud/current/client-apps/optimizing/latency.html and @KetanA for persisting throughout the experiment.