The problem of this approach is that there is no orchestration of the work, and this means that in case of failure, a a client would have to iterate over the list of nodes until it finds one which is successfully handling the request.
Describe the solution you'd like
before sending the job to the workers and select RR one by one from the list, we want to have a pre-orchestration phase by using a separate communication channel ( a gossip pub/sub channel) which allows to the nodes to syncronize. The protocol should be the following:
A client sends a work request to the pub sub channel (state exchange channel)
The client receives answers from the workers, and selects the worker (the worker that replies are the only ones that can fullfill the request). It keeps a list of the workers that successfully answered
The client tries to deal the workers, verifies that can connect
The client sends the job
In case of failure, the client goes back to 3 by selecting another worker (Least resource used)
The client acknowledges the work has been done and broadcast that the job was executed successfully, or refused to the pub/sub gossip channel for state exchange
When we have #496 in, we can:
have the ledger recording the work has been done
the ledger selects the work in step 2, and automatically can have a record of the reward for #382
Describe alternatives you've considered
In the long run, we want the ledger to select the workers that can fullfill a user request so we can control fairness (step 2 and 6 covered by the ledger). This card is skewed to only provide a gossip protocol for orchestration before we jump into a full implementation
Additional context
Adding the diagram that was built from the brainstorming session:
Acceptance criteria
[ ] We have a state exchange gossip pub/sub between all the nodes
[ ] The client and the workers follow the protocol defined above with the state exchange pub/sub channel to distribute the work
[ ] Follow-up card about incorporating #496 and having the ledger select the worker for the client
mudler
commented
3 weeks ago
Is your feature request related to a problem? Please describe. With this PR: https://github.com/masa-finance/masa-oracle/pull/504 and https://github.com/masa-finance/roadmap/issues/69 we are introducing a way to select workers directly from the list of the nodes. After that, we dial in with libp2p the nodes, and we send over the work.
The problem of this approach is that there is no orchestration of the work, and this means that in case of failure, a a client would have to iterate over the list of nodes until it finds one which is successfully handling the request.
Describe the solution you'd like before sending the job to the workers and select RR one by one from the list, we want to have a pre-orchestration phase by using a separate communication channel ( a gossip pub/sub channel) which allows to the nodes to syncronize. The protocol should be the following:
pub/subgossip channel for state exchangeWhen we have #496 in, we can:
Describe alternatives you've considered
In the long run, we want the
ledgerto select the workers that can fullfill a user request so we can control fairness (step 2 and 6 covered by the ledger). This card is skewed to only provide a gossip protocol for orchestration before we jump into a full implementationAdditional context
Adding the diagram that was built from the brainstorming session:
Acceptance criteria