Route messages based on condition

[yanivbh1]

Motivation

This feature, in other words, can be called an "Event Router." It enables routing messages to different consumers based on payload and predefined rules. Many security-related use cases require that feature that can eventually replace ACLs or at-least act as another layer of security.

Suggested Memphis implementation

A modular router that can be configured at the station level where users can define routing rules based on message payload and for specific sinks or consumers. For example: Sink messages that contain specific keys, fields, or headers in their payload to Elastic search and the rest to other consumers.

Asked by

• Yehuda Korotkin

[pbell23]

Hello,

Very interested in this feature. As it is not available for now, I have found two possible solutions.

First, have consumers in different groups so messages are sent accros all groups then consumers just ignore those which are not intented for it.

Second, create different stations for different messages purposes.

What do you think is the best solution ?

Do you have an idea when this feature will be implemented ?

Thank you.

[yanivbh1]

Hey @pbell23 , I think the first question should be who will absorb the back pressure. The client or the broker. Who will do the dropping? It's still under research, ETA is by the end of Q1 2023.

[TroyKomodo]

Is this feature already implemented? There seems to be docs about it over here https://docs.memphis.dev/memphis/memphis/comparisons/apache-kafka-vs-memphis#message-routing

[yanivbh1]

Hey @TroyKomodo , Currently available for design partners before going public. Will be released soon.

[TroyKomodo]

Hi @yanivbh1, Is there a timeline for this feature?

[yanivbh1]

@TroyKomodo should be around Sep. We can discuss in private and see if it makes sense to include you in our partners list and expose it to your company earlier.

[TroyKomodo]

Sure, feel free to reach me troy at scuffle dot tv