Purpose of xk6-kafka

[Badjangas]

Hi there Mostafa, hope you're well :) I'm sure this may very well seem like a very stupid question but I just can't wrap my head around it – I found this k6 extension while searching for something that would help me in performance testing my application, more specifically, my application's kafka consumer. I've read this repo's readme guide and played a little bit with the extension and I have everything up and running – what I don't seem to understand is what is the extension actually testing? Is it my application's message consumption rate or something else? I'm pointing xk6-kafka reader at the topic my applications subscribes from btw. Since I'm creating a reader object that's consuming from the topic, I'm not really evaluating how is my application's consumer performing right? This is my confusion. And if this is true maybe xk6-kafka is not what I really need, am I wrong? Please could you help clarify please? Thanks and thank you for developing this :) it's been fun exploring it.

(edit) PS: I know the repo literally begins with the following paragraphs:

The xk6-kafka project is a k6 extension that enables k6 users to load test Apache Kafka using a producer and possibly a consumer for debugging.

The real purpose of this extension is to test the system you meticulously designed to use Apache Kafka. So, you can test your consumers, hence your system, by auto-generating messages and sending them to your system via Apache Kafka.

But in what sense does it help me in gathering information on how my application is performing if I'm calling the reader object to consume the messages?

[mostafa]

Hey @Badjangas,

You don't have to use the Reader object and it is just there for certain case, in which someone might want to test their Kafka deployment or test a certain producer in their platform.

Simply use the Writer to produce messages, and wait for your system on the other end to consume them. While doing so, monitor the metrics produced on both sides to measure if your system is performant enough and meets your expectations or not.

Note that xk6-kafka emits a lot of metrics, and you can send those metrics to other platforms for visualization and monitoring, like Prometheus or any other real-time results output that is supported by k6.

[Badjangas]

Thanks for the answer @mostafa :) never expected to get an answer so fast! Appreciated it 🤝

Simply use the Writer to produce messages, and wait for your system on the other end to consume them.

Yeah so basically what I was suspecting yeah although the Reader object wasn't making much sense in my mind if used for this purpose. Yeah it's gonna e tricky to integrate all this things as I've never done it :s (nor worked with Prometheus)

Basically I believe I'll need to query my db and check how many events have been stored and processed successfully (ie not in an errored status) against the number os messages consumed. But what I'm finding tricky is how to measure the possible delta between messages when the load is high (assuming that with higher loads my system begins to choke a bit)...

Again, thank you for your answer :)

[mostafa]

@Badjangas

There are many ways to observe how your application works, including querying database for number of rows processed. However, it'd be better that you add observability to your application, so that you know what is going on in your consumers, your DB writes/reads, etc. This way your application reports metrics to the same metrics database (for example, Prometheus) and you can correlate the incoming load towards your Kafka cluster (no. of messages, lag, etc.) with those metrics your application emitted. Something like this:

flowchart TB
    xk6-kafka-- produces messages-->kafka["Apache Kafka"]
    xk6-kafka-- produces metrics-->Prometheus
    kafka-- produces metrics -->Prometheus
    kafka-- consumes messages-->platform["Your consumers in your platform"]
    platform-- produces metrics-->Prometheus
    Grafana-- query metrics-->Prometheus
    user-- visualize and correlate metrics -->Grafana

[mostafa]

Hey,

I suppose your question is answered, I'll close this issue. If you have further questions related to the same issue, please re-open this, otherwise open a new one.