SPIKE: Worker selection heuristics

mudler commented 5 months ago

We have already an heuristic for selecting a worker in a distributed manner (see #11), however currently all the workers will receive the work request, and we have hundreds of workers for each of the request that the protocol has to fullfill.

This card is to optimize the heuristic to pick a worker for a specific request, or alternatively a subset of the workers. Theoretically, we can also let specify by API the number of workers that must be involved in the answer.

This card is a spike to understand what options we have and what's possible to achieve - especially to get a high level list of options that we can consider before starting implementation.

Acceptance criteria

[x] We have gained understanding on how worker selection heuristic currently are handled in the masa-oracle code (share relevant links here)
[x] We have a list of options that aims to improve the worker selection
[x] Every option has a pro/cons so we can take an informed decision

mudler commented 4 months ago

possibly this is for a v1 - we need to figure out if there is a worker selection criteria and if that's enough for a v0

Luka-Loncar commented 3 months ago

If twitter account 429's the request will not be passed to the next worker. We probably need a new ticket for this thing. @mudler @teslashibe @restevens402

restevens402 commented 3 months ago

Description of possible implementation

Introduce a mechanism for tracking and checking node capabilities to ensure work is only dispatched to nodes equipped to handle it. Limit the number of nodes that work is dispatched to. This will prevent unnecessary work dispatch and improve system efficiency. If a node hits request limits put it in timeout.

Tasks

[x] Before processing work, handlers should check if the receiving node has the required capabilities.
[x] If a node lacks the necessary capabilities, the handler should log an appropriate message and cease further processing.
[x] use NodeData for tracking the capabilities of each node.
[x] Have an upper limit for the number of nodes to send work requests
[x] Add feature timeout timestamps to nodeData and helper methods to do timeout checks

Acceptance Criteria

Handlers check node capabilities before processing and refrain from processing if capabilities are lacking.
Utilize the flags on the NodeData to determine if a node appears capable of handling the specific type of request.
If the local node is capable of handling the request, prefer local processing and do not send to the network unless there is an error condition.
Only send N number of network requests and accept the first one to return. N should be defined by the team and set in the application configuration.
If the local node returns a 429 error, add a timestamp to the nodeData and do not use a node that has a timestamp within N minutes. Add helper methods on NodeData to make this easy from the worker code.

teslashibe commented 3 months ago

@mudler this is being worked on here: https://github.com/masa-finance/masa-oracle/pull/489

teslashibe commented 3 months ago

@mudler I moved this to done and added to 0.6.0 release

mudler commented 3 months ago

@mudler I moved this to done and added to 0.6.0 release

Shall we have a follow-up for potential improvements before closing? or are we considering the story completely closed with https://github.com/masa-finance/masa-oracle/pull/489 ? (moved in review as the PR isn't yet merged)

mudler commented 2 months ago

we are good with the solution provided with https://github.com/masa-finance/masa-oracle/pull/489 for now, however, we might need to get back to this by seeing more tests in the wild and probably when picking up #66

masa-finance / roadmap