Show message headers

[mawek]

Is there some way how to also show message headers? It would also be nice to be able to filter by them.

[wlezzar]

Hi @mawek . Thanks for raising the issue : )

This is definitely a good idea. It's not possible currently to do so but it shouldn't be hard to implement this feature. We will work on it and try to make this available in the next release.

Can you describe a bit more in detail the use case? Would you like to filter on the presence / absence of a header? Or do you need to be able to perform full jmespath / jq queries on headers content?

[mawek]

Thanks for looking into it. It is really hard to find kafka client that also works with headers - even native kafka client doesn't support them. When working with kafka, we put "business" data into the message, and "technical metadata" go to headers (like trace_id, event_id, user_id,...). It would be nice to support also filtering by them.

I also notice, that output doesn't contain partition key - nor does it contain event name.

We use kafka for event driven architecture, so we put different events (with different schemas) to single topic (related to single domain). So for example "user" topic contains events like UserNameUpdated, UserRolesChanged, etc.

When investigating issue we often want to peek inside of "user" topic and find some specific event - mostly we filter by event name, partition key (which is in this example user id) or by event id (stored in headers), or sometimes we know just partition and offset event is stored on.

Currently we use our own JMX bean for that, but I'm looking for a tool that could replace it.

[wlezzar]

Super interesting feedback.

I see no blocker in implementing this in Zoe. I will iterate on this and try to implement a first version. I will ask you for feedback on this thread.

[wlezzar]

Hi @mawek . So basically I have a first draft here.

Basically, this would add 2 options in the CLI to the topics consume command:

• --filter-meta that allows users to express jq / jmespath filters on headers, partitions, keys and other record metadata.
• --with-meta that prints the consumed records with their metadata (offset, partition, headers, etc.)

Does that sound good?

[mawek]

That looks cool 👍 Not sure how will it play with table output option - I suppose it will be part of meta column? Is it correct? One more question - does key also support avro schema?

[wlezzar]

Thanks for the feedback. Yes keys support avro schemas in the consumption. If the key is an avro record, it will appear in zoe's output as a json string.

[G3zz]

This is exactly what I need! Any idea when it will be released? Thanks

[wlezzar]

Hey @G3zz . I'm gonna trigger a release to make this feature available and then close the issue. I will keep you updated.

[wlezzar]

Ok you can download the 0.25.0 release and this feature should be there : )