search

mailgun / kafka-pixy

gRPC/REST proxy for Kafka
Apache License 2.0
768 stars 119 forks source link

Question: Perf test metrics #177

Open eajazali87 opened 4 years ago

eajazali87 commented 4 years ago

Hi,

We are able to produce and consume messages from the kafka cluster through the kafka-pixy. I wanted to do some performance tests around this.

Simple Use Case:

  1. There are 1M messages in a topic in the Kafka cluster
  2. Consume 1M messages from topic(kafka cluster) through the kafka-pixy consumer client
  3. Report performance metrics.

NOTE: There is kafka-consumer-perf-test.sh script that is available in the kafka library kafka-consumer-perf-test.sh --broker-list localhost:9092 --topic dc-topic-1 --messages 1000000

If there is something like this or any pointers/scripts around this would be really really helpful

horkhe commented 4 years ago

@eajazali87 in the release package you can find testconsumer and testproducer binaries provided precisely for load testing. E.g.:

./testconsumer --addr localhost:19091 --topic demo --count 1 --group foo
./testproducer --addr localhost:19091 --topic demo --count 1000000 --threads 1000 --sync --size 1024
./testconsumer --addr localhost:19091 --topic demo --count 1000000 --threads 10 --group foo

The very first testconsumer call is there to initialise the consumer group foo partition offsets.

eajazali87 commented 4 years ago

Thanks @horkhe when I run the first script, I get this error. Any way to get rid of this?

My GO version: go version go1.12.9 darwin/amd64

eajazalimahaboobbasha:kafka-pixy-v0.17.0-darwin-amd64 emahaboobbasha$ export GRPC_GO_LOG_SEVERITY_LEVEL=info
eajazalimahaboobbasha:kafka-pixy-v0.17.0-darwin-amd64 emahaboobbasha$ ./testconsumer --addr localhost:19091 --topic main_topic --count 1 --group 20
INFO: 2019/12/05 14:38:56 Channelz: socket options are not supported on non-linux os and appengine.
INFO: 2019/12/05 14:38:56 parsed scheme: ""
INFO: 2019/12/05 14:38:56 scheme "" not registered, fallback to default scheme
INFO: 2019/12/05 14:38:56 ccResolverWrapper: sending new addresses to cc: [{localhost:19091 0  <nil>}]
INFO: 2019/12/05 14:38:56 ClientConn switching balancer to "pick_first"
INFO: 2019/12/05 14:38:56 pickfirstBalancer: HandleSubConnStateChange: 0xc00006b110, CONNECTING
INFO: 2019/12/05 14:38:56 blockingPicker: the picked transport is not ready, loop back to repick
INFO: 2019/12/05 14:38:56 pickfirstBalancer: HandleSubConnStateChange: 0xc00006b110, READY
INFO: 2019/12/05 14:38:56 transport: loopyWriter.run returning. connection error: desc = "transport is closing"
INFO: 2019/12/05 14:38:56 pickfirstBalancer: HandleSubConnStateChange: 0xc00006b110, TRANSIENT_FAILURE
INFO: 2019/12/05 14:38:56 pickfirstBalancer: HandleSubConnStateChange: 0xc00006b110, CONNECTING
INFO: 2019/12/05 14:38:56 pickfirstBalancer: HandleSubConnStateChange: 0xc00006b110, TRANSIENT_FAILURE
panic: failed to consume first: rpc error: code = Unavailable desc = transport is closing
goroutine 36 [running]:
main.main.func1.1(0xc00013a040, 0x1536600, 0xc000158000, 0xc00006b070, 0xc000146000, 0x1, 0x0)
    /Users/maxim/src/github.com/mailgun/kafka-pixy/tools/testconsumer/testconsumer.go:75 +0x858
created by main.main.func1
    /Users/maxim/src/github.com/mailgun/kafka-pixy/tools/testconsumer/testconsumer.go:57 +0xf1
eajazalimahaboobbasha:kafka-pixy-v0.17.0-darwin-amd64 emahaboobbasha$
eajazali87 commented 4 years ago

Figured out the problem, the local Kafka-pixy expects TLS where as the test consumer is trying plaintext. Once I set this to the same type, it worked

image
horkhe commented 4 years ago

I am glad that you figured it out. It would be great if in the end you could share the results of your load testing. Maybe post them in this ticket or in the README.md.

emahaboobbasha commented 4 years ago

sure I will do that @horkhe thanks for your support.