ShiningRush / fastflow

A lightweight, high-performance distributed workflow framework
MIT License
348 stars 80 forks source link

RFC: support pass unique key to Keeper #25

Open Ehco1996 opened 1 year ago

Ehco1996 commented 1 year ago

Hi, first of all thank you for the library!

In the process of trying to use it, we found that because the worker's unique id is achieved through sonyflake, the worker requires an integer between 0 and 255 to achieve the unique id when it is started

https://github.com/ShiningRush/fastflow/blob/1afa59285ab304f913b0c8de8e044bcad86af5f6/keeper/keeper.go#L14

In practice, this restriction would lead to reliance on external storage to generate the unique integer in multi-process situations, so I propose that the global unique key can be passed directly from the caller and the keeper component no longer generates the unique key based on number

There are two advantages to this change

If you agree to this change, I will provide PR in my free time

thanks

ShiningRush commented 1 year ago

Hi echo, thanks for your proposal, basically I agree with it. I want declare two points:

  1. Worker key limited 0~255 is incorrect, sonyflake is support to use uint16(0~65536) as machine_id, this should be a bug of fastflow
  2. using worker key suffix as unique number is very convenient in k8s, because you can deploy your application by stateful set, it can auto generate pod name like stateful-set-0, stateful-set-1 ...

however, support a new way to generate id is LGTM. I think we can decouple id generator: