[Feature] Event Filter

[AhahaGe]

Search before asking

• [X] I had searched in the issues and found no similar issues.

Feature Request

1. Scenario

Traditional, when consumer want to filter interested messages, consumer need to rely on specified MQ client and code upon it to accomplish. But when it comes to apache EventMesh, users only need to submit customized expressions which defines message filter logic to EventMesh, EventMesh will do the filter and return filtered messages to users.

2. Implementation

2.1 Implementation Plan One

Do stream processing in EventMesh, it's MQ independent, but have some impact in performance. And we can maintain the expression engine ourself. Concrete implementation, define a set of EventMesh api to integrate and implement filter rule engine, combine with EQL and Json path. And we can also consider integrating the research rules engine (DataSonnet).

2.2 Implementation Plan Two

Bind to RocketMq, use RocketMq as engine to realize filter. Concrete implementation, implement openmessaging-api interface, and use RocketMq to realize filter ability.

2.3 Implementation Conclusion

At present, it is inclined to plan 2.1. The ability of filter has nothing to do with specific MQ components. The specific plan still needs to be investigated.

2.4 related problems:

1. Can Kafka originally support filter?
2. Does RocketMq's filter support multiple thread, currently support.
3. Consider multi-tenant segregation.
4. The storage problem of rules supports plug-in implementation.
5. How about integrate the rules engine, such as Drools or EasyRule?

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

[ruanwenjun]

Plan 2.1 look good to me. I think the performance will not be impacted a lot unless we need to do some join operation.

[xwm1992]

Do we need to integrate the rules engine, such as Drools or EasyRule ?

[AhahaGe]

Do we need to integrate the rules engine, such as Drools or EasyRule ?

Sounds good. We can discuss later and I add it into problem list.

[jinrongluo]

I have a quick look at Drools and EasyRule, their rules are not for filtering. I think we need the simple rules like RocketMQ filters

https://rocketmq.apache.org/docs/filter-by-sql92-example/

for example: $.subject == 'myservice'

We can implement our own simple Rule engine to process this rules using Parse Tree (https://en.wikipedia.org/wiki/Parse_tree) and JsonPath SDK

Concrete Implementation:

1. Parse the Filter Expression into Parse Tree
2. Evaluate each node of the Parse Tree using JsonPath, and Java API

[AhahaGe]

Do u mean just like a sql where filter, where topic = 'myservice' and name='a' or age = '1', and what we need is a engine like a sql parse engine, and what eventmesh need to do is filter message stream using these where conditions as design pattern responsibility chain in a filter plugin to filter each message.

[jinrongluo]

I think SQL where syntax is good enough. What do other people think?

Alex

On Apr 14, 2022, at 9:27 PM, AhahaGe @.***> wrote:

 Do u mean just like a sql where filter, where topic = 'myservice' and name='a' or age = '1', and what we need is a engine like a sql parse engine?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[xwm1992]

I have a quick look at Drools and EasyRule, their rules are not for filtering. I think we need the simple rules like RocketMQ filters

https://rocketmq.apache.org/docs/filter-by-sql92-example/

for example: $.subject == 'myservice'

We can implement our own simple Rule engine to process this rules using Parse Tree (https://en.wikipedia.org/wiki/Parse_tree) and JsonPath SDK

Concrete Implementation:

1. Parse the Filter Expression into Parse Tree
2. Evaluate each node of the Parse Tree using JsonPath, and Java API

This is a good proposal，may be we can learn more about the Parse Tree and JsonPath SDK. If we want to implement our own rule engine another direction to consider is implement this under the JSR 94, looking forward to other people opinion.

[Pil0tXia]

There are some good examples that can serve as references for the UI of the filter. For instance, Cloudflare's Transform Rules: