Filter messages by header key

[fparareda]

Is your feature request related to a problem? Please describe. If we are consuming a topic with a huge amount of events and each of those events has a key in the header to differentiate one between the others, should be necessary a way to filter those messages, without deserialize message to access headers. Improving the filtering considerable.

Describe the solution you'd like Filter messages without any deserialize payload to speed up the filtering.

• Spring is doing it here and here and the documentation here
• Kafka connect is giving the possibility to do it in Connectors here
• There is an issue open for the same problem with .NET in Confluent

Additional context An example would be: I am logging every action in a topic, but inside of it there are some actions (CRUD): Creation, Read, Update and Deletion. If this action is in a key in the header (aka action: creation), with this feature, those subscribers that only wants to be informed once there is an event of creation, will be discarding other events without even deserializing the payload (what it's the most expensive part to deserialize).

[OneCricketeer]

AFAIK, Kafka itself has no way to even index a topic by headers, therefore a client wouldn't improve anything there, and so it must consume and deserialize all records with their headers.

Spring / Connect API is still consuming every event, and only delivering those that match the predicate into the process functions.

You're not required to deserialize the events from KafkaJS; the key and value are always Buffer types