[Feature] Design of EventMesh Function

xwm1992 commented 7 months ago

Search before asking

[X] I had searched in the issues and found no similar issues.

Feature Request

Design of EventMesh Function

For the existing EventMesh, the connector, eventmesh runtime and other modules can be jointly abstracted into a runtime component. This component can be used as connector runtime\mesh runtime\function runtime, etc. The specific component structure and interaction are shown in the following figure:

Admin

The admin component includes the task scheduling function, which can assign tasks to the connector and manage the connector life cycle. The admin service will be registered in the Registry for connector service discovery. The admin and connector use grpc to communicate.

Connector

The connector runs in the connector runtime. After the connector is started, the Admin service will be discovered from the Registry. After selecting the admin service from the available admin service list, it will report the heartbeat to the admin service and carry the currently assigned task information. There is no task information in the initial state. Admin will assign task to the connector based on the received heartbeat and task information, and the tasks will be recorded in the DB. After receiving the task, the connector performs task initialization and loads the corresponding connector plug-in.

Function

Function runs in function runtime. Function mainly provides filter/transformer/compute capabilities. The admin can configure function-related rules. Function needs to communicate through Mesh.

Mesh

Mesh runs on mesh runtime. Mesh provides routing, pub/sub, and serverless automatic expansion and contraction capabilities. Mesh runtime integrates netty server by default. All traffic will be converged and summarized in mesh. Automatic expansion and contraction can be achieved by integrating knative's serving capabilities which needs to be deployed under k8s. For k8s deployment, as shown below, different components to be run are defined through yaml. After receiving the request, the EventMesh Operator will create k8s resources for different components.

Storage

The connector can integrate a variety of storage plug-ins and uses memory storage by default.

EventMesh Function设计

对于现有的EventMesh来说，可以将connector、eventmesh runtime等模块共同抽象出一个runtime组件，这个组件可以作为connector runtime\mesh runtime\function runtime等等，具体的组件结构与交互如下图所示：

Admin

admin组件包含了任务调度的功能，可以对connector分配任务, 管理connector生命周期, admin将自己这个服务上报到Registry供Connector进行服务发现, admin与connector之间采用grpc方式进行通信。

Connector

connector运行于connector runtime, connector启动后会从Registry发现Admin服务，从可用的admin服务列表中选择admin服务后，向该admin服务上报心跳，并携带当前分配的任务信息，初始状态无任务信息，admin会根据收到的心跳以及任务信息来分配任务，并将任务记录到DB中。connector接收到任务后，执行任务初始化，加载对应的connector插件。

Function

function运行于function runtime, function主要提供filter/transformer/compute的能力，可以由admin来配置function相关的规则，function需通过Mesh进行通信交互。

Mesh

mesh运行于mesh runtime, mesh提供了routing、pub/sub以及serverless自动扩缩容的能力, mesh runtime默认集成了netty server, 所有的流量会收敛汇总于mesh, 通过集成knative的serving能力实现自动扩缩容, 同时需要在k8s下部署, 对于k8s部署如下图, 通过yaml定义要运行的不同组件, EventMesh Operator收到请求后会对不同组件创建k8s资源。

Storage

connector可集成多种storage插件，默认采用内存存储。

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[ ] I agree to follow this project's Code of Conduct *

Pil0tXia commented 7 months ago

Registry is the current meta module of eventmesh

Will we change the name to "Registry" to refer to the registry center in the future? Or will we continue with the current "Meta" name?

qqeasonchen commented 7 months ago

Let's fire~

KevinClair commented 7 months ago

Is there a specific division of labor in this area currently? I want to participate

Pil0tXia commented 7 months ago

@KevinClair

Is there a specific division of labor in this area currently?

I think we haven't have one yet. It will be nice if you are willing to pick up anything that interests you and be the leading one~.

xwm1992 commented 7 months ago

Is there a specific division of labor in this area currently? I want to participate

We will first split the entire project into different modules, and then carry out related division of labor. We will raise different issues according to the modules, so you can continue to pay attention.

Ish1yu commented 7 months ago

I Voting named "Registry"，And look forward to more

rottenmu commented 7 months ago

Would it be more appropriate to think of the registry as a metadata service？

xwm1992 commented 7 months ago

Thank you everyone for the discussion, I made relevant modifications to the above description. I changed the Registry in the picture to Meta, keeping it consistent with the modules of our current project to avoid ambiguity.

rottenmu commented 7 months ago

May I'm do somthing ?

xwm1992 commented 7 months ago

May I'm do somthing ?

We will first split the entire project into different modules, and then carry out related division of labor. We will raise different issues according to the modules, so you can continue to pay attention.

rottenmu commented 7 months ago

Ok

Pil0tXia commented 7 months ago

One problem with the existing EventMesh SDK is that the user can only specify the IP address to which a particular EventMesh Runtime node is connected.

In the new architecture, which IP address needs to be filled in the SDK? If the address of the Mesh Runtime is filled in, how will the user be able to utilize the failover functionality when the specific Mesh Runtime address goes offline?

mxsm commented 7 months ago

There are currently custom data protocols, Grpc and CloudEvents protocols, which are the formats of external protocols. The data flow within the EventMesh internal runtime is also accompanied by protocol conversion. Multiple protocol conversions lead to a decline in performance. Whether this architecture change can redesign the data protocol. The general idea is:

Keep only one within the runtime, regardless of where it is stored.
Implement the current mainstream protocols such as Grpc, cloudenvents, MQTT, etc. externally.
Provide an extensible interface for external protocol access, which can access external user-defined protocols.

qqeasonchen commented 6 months ago

There are currently custom data protocols, Grpc and CloudEvents protocols, which are the formats of external protocols. The data flow within the EventMesh internal runtime is also accompanied by protocol conversion. Multiple protocol conversions lead to a decline in performance. Whether this architecture change can redesign the data protocol. The general idea is:

Keep only one within the runtime, regardless of where it is stored.

Implement the current mainstream protocols such as Grpc, cloudenvents, MQTT, etc. externally.

Provide an extensible interface for external protocol access, which can access external user-defined protocols.

agree~

apache / eventmesh