One tool I use all the time for debugging is kcat -b <broker_address> -Ct <topic_name>. That essentially does tail -f <topic_name> - it lets me watch the end of the topic in real time to see what data's coming in. The quix CLI current makes this harder, but in a way I think reveals a couple of useful new features.
So imagine I'm running a pipeline with quix pipeline up and I want to watch a topic. How do I do it? How do I find the broker address and the topic name?
Where's the Broker Address?
It's probablylocalhost:19092. But I think quix pipeline status should tell me explicitly.
What's the topic name?
This is findable - it's going to be in the code, or in app.yaml, or quix.yaml. But that's a lot to look through. quix topic list or quix pipeline topics list would be very useful.
Do we even need kcat?
kcat -Ct is just a consumer. How about we add a basic consumer into the CLI, so you can call quix topic tail <topic_name>? That removes the need for another tool and makes quix CLI even more useful. (Actually, between you and me, kcat is not the most user-friendly tool in the world. I think we could leapfrog ahead of it. 😉)
One more thought - add it to the pipeline view
quix pipeline view generates a nice mermaid diagram from a markdown document. How about we add quix topic list's output as a markdown table?
For reference
Here's another useful kcat command we can take inspiration from. List topics:
quix pipeline status # Now shows the broker address.
quix pipeline view # Now shows topics and their metadata as well.
quix topic list
quix topic tail <topic_name>
One tool I use all the time for debugging is
kcat -b <broker_address> -Ct <topic_name>
. That essentially doestail -f <topic_name>
- it lets me watch the end of the topic in real time to see what data's coming in. The quix CLI current makes this harder, but in a way I think reveals a couple of useful new features.So imagine I'm running a pipeline with
quix pipeline up
and I want to watch a topic. How do I do it? How do I find the broker address and the topic name?Where's the Broker Address?
It's probably
localhost:19092
. But I thinkquix pipeline status
should tell me explicitly.What's the topic name?
This is findable - it's going to be in the code, or in
app.yaml
, orquix.yaml
. But that's a lot to look through.quix topic list
orquix pipeline topics list
would be very useful.Do we even need
kcat
?kcat -Ct
is just a consumer. How about we add a basic consumer into the CLI, so you can callquix topic tail <topic_name>
? That removes the need for another tool and makes quix CLI even more useful. (Actually, between you and me,kcat
is not the most user-friendly tool in the world. I think we could leapfrog ahead of it. 😉)One more thought - add it to the pipeline view
quix pipeline view
generates a nice mermaid diagram from a markdown document. How about we addquix topic list
's output as a markdown table?For reference
Here's another useful
kcat
command we can take inspiration from. List topics:Related: http://blog.jenkster.com/2022/10/setting-up-kcat-config.html
To Summarize
I think we should add: