Looking for feedback on a testing tool for uReplicator

[rantav]

This isn't an actual issue, but I wasn't sure what's the best way to get in touch with the authors so here we are...

We're evaluating uReplicator at AppsFlyer and in this process I created a test framework which basically provisions two kafka clusters in two different datacenters (us-east and eu-west) and runs uReplicator and various test and failure cases to see what happens.

It uses:

• A special home-built golang producer and consumer (for message validation, latency measurement and load generation)
• Kafka (2.1.0 but other versions can be swapped in)
• Latest uReplicator
• Kubernetes for automation
• Prometheus and Grafana for measurement

Would love to get your feedback.

Repo: https://github.com/AppsFlyer/kafka-mirror-tester Design: https://github.com/AppsFlyer/kafka-mirror-tester/blob/master/README.md Results: https://github.com/AppsFlyer/kafka-mirror-tester/blob/master/results.md

Thank you

[yangy0000]

@xhl1988

[rantav]

@xhl1988 @yangy0000 did you get a chance to look into this? I'd appreciate your feedback, thank you!

[yangy0000]

@rantav Thank you for sharing testing tool, it's the first time to see the performance benchmark result in the cloud environment. We have many ongoing improvements for ureplicator including support header and security. We could use this as a performance benchmark.

[yangy0000]

Running this testing cluster on Cloud can be costly, do you think this can be easily adjusted to running on the local environment with a smaller cluster?

[rantav]

do you think this can be easily adjusted to running on the local environment with a smaller cluster?

It certainly can. As a matter of fact I initially developed on a local kubernetes (minikube) cluster and it was able to run the entire stack.

For my use case it was crucial to test high throughput, high scale and real world cross-continent latency and reliability so I did this. But it can be scaled down with some kubernetes love (somewhat trivial, but still work)