alfetahe / process-hub

Distributed processes manager and global process registry
GNU General Public License v3.0
62 stars 2 forks source link

Preemptive Synchronization Strategy #5

Open sleipnir opened 2 months ago

sleipnir commented 2 months ago

Hello, me again. ProcessHub declares two synchronization strategies, PubSub and Gossip. We know that PubSub is better suited when we have fewer nodes connected to the cluster and that Gossip has better performance for larger clusters. My question is: How difficult would it be to implement a Preemptive strategy, where the size of the cluster doesn't matter because ProcessHub will adapt to the strategy that best makes sense depending on the size of the cluster? This would be very interesting for dynamic clusters, where they start small but during their life cycle more and more nodes are added to them.

I imagine that the user could define the limits so that the strategy could know when the number would be considered small and therefore PubSub user and when it would be considered large and the strategy would have to switch to Gossip.

What do you think @alfetahe?

alfetahe commented 2 months ago

Hi,

I think this could be a valuable addition where the system replaces strategies dynamically based on certain parameters and rules. Currently, all strategies are stored in the ETS table. When some code needs to trigger a callback on them, it pulls the strategy struct from the ETS table and uses the base protocol method to call the function. The protocol itself dispatches to the correct module based on the struct. We could easily swap the structs in the ETS table dynamically.

However, I do have some concerns about hook handlers that register with strategy structs, as those would need to be updated as well.

Regarding the Gossip strategy, it currently works for propagating information about new processes or process removal in most cases, but there are a few parts of the code that still rely on the PubSub mechanism. These should be addressed in future releases.

Overall, I like the idea and believe it should be implemented in the future.

sleipnir commented 2 months ago

Awesome @alfetahe

alfetahe commented 1 month ago

I will keep this issue open so it can be dealt with in the future releases.