[ISSUE #4852] Support Jraft algorithm as meta storage - Githubissues

apache / eventmesh

EventMesh is a new generation serverless event middleware for building distributed event-driven applications.

https://eventmesh.apache.org/

Apache License 2.0

1.56k stars 618 forks source link

[ISSUE #4852] Support Jraft algorithm as meta storage #4862

Closed karsonto closed 3 days ago

karsonto commented 1 month ago

Fixes #4852

Motivation

Support Jraft algorithm as meta storage

Modifications Removed unused variable responseEventMeshCommand in SendSyncMessageProcessor#processRequest

Documentation New Configuration Information Explanation

eventMesh.metaStorage.raft.dataPath=/tmp/server1 // Path to local snapshot storage for raft data
eventMesh.metaStorage.raft.self=127.0.0.1:9091 // Raft protocol local ip and port
eventMesh.metaStorage.raft.members=127.0.0.1:9092,127.0.0.1:9093 // Raft members node ip and port
eventMesh.metaStorage.raft.electionTimeout=5 // Raft Election Leader Timeout second
eventMesh.metaStorage.raft.snapshotInterval=30 // Raft snapshot Interval Timeout second
eventMesh.metaStorage.raft.refreshLeaderInterval=3 // Raft refresh Leader node Interval Timeout second

Pil0tXia commented 1 month ago

Please be noted that you're linking an irrelavant issue.

Pil0tXia commented 1 month ago

May you please introduce briefly that how JRaft stores metadata and how to retrieve them (such as subscription info) from clients (such as EventMesh Dashboard or user)?

karsonto commented 1 month ago

May you please introduce briefly that how JRaft stores metadata and how to retrieve them (such as subscription info) from clients (such as EventMesh Dashboard or user)?

My understanding, Jraft is to synchronise the data to the cluster internal machine, each machine above the data is consistent, this PR is to save the data in memory, every once in a while will snapshot the data to the local disk, start the server will snapshot the data loaded into memory, EventMesh Runtime needs to provide restful api in order to allow EventMesh Dashboard to get the data saved by the jraft!

Pil0tXia commented 1 month ago

EventMesh Runtime needs to provide restful api in order to allow EventMesh Dashboard to get the data saved by the jraft!

May you please indicate which class to interact with in order to get JRaft data?

karsonto commented 1 month ago

EventMesh Runtime needs to provide restful api in order to allow EventMesh Dashboard to get the data saved by the jraft!

May you please indicate which class to interact with in order to get JRaft data?

You can get MetaStorage in the Runtime, and then get MetaService, which is the implementation class of RaftMetaService.

Pil0tXia commented 1 month ago

Please add known dependencies~

Pil0tXia commented 1 month ago

Implemented JRaft Feature in this PR

In the current implementation of this PR, if a new node is added to the cluster, users need to add the new node to the eventmesh.properties of the existing nodes before it can be added to the cluster.

Hopefully you can implement CliService in a next PR to support service registration when automatic scaling up and down of the cluster. Because the complete list of online nodes of the cluster cannot be maintained in real time.

After it's implemented, the newly added node can connect to any online node in the cluster. If connected to a follower, the current leader will be returned to new node by the follower.

karsonto commented 1 month ago

If you get bugs during use, I will continue to improve it later, thanks!

karsonto commented 6 days ago

@mxsm Done,please review.