Use gossip to co-ordinate worker selection

[mudler]

Is your feature request related to a problem? Please describe. With this PR: https://github.com/masa-finance/masa-oracle/pull/504 and https://github.com/masa-finance/roadmap/issues/69 we are introducing a way to select workers directly from the list of the nodes. After that, we dial in with libp2p the nodes, and we send over the work.

The problem of this approach is that there is no orchestration of the work, and this means that in case of failure, a a client would have to iterate over the list of nodes until it finds one which is successfully handling the request.

Describe the solution you'd like before sending the job to the workers and select RR one by one from the list, we want to have a pre-orchestration phase by using a separate communication channel ( a gossip pub/sub channel) which allows to the nodes to syncronize. The protocol should be the following:

1. A client sends a work request to the pub sub channel (state exchange channel)
2. The client receives answers from the workers, and selects the worker (the worker that replies are the only ones that can fullfill the request). It keeps a list of the workers that successfully answered
3. The client tries to deal the workers, verifies that can connect
4. The client sends the job
5. In case of failure, the client goes back to 3 by selecting another worker (Least resource used)
6. The client acknowledges the work has been done and broadcast that the job was executed successfully, or refused to the pub/sub gossip channel for state exchange

When we have #496 in, we can:

8. have the ledger recording the work has been done
9. the ledger selects the work in step 2, and automatically can have a record of the reward for #382

Describe alternatives you've considered

In the long run, we want the ledger to select the workers that can fullfill a user request so we can control fairness (step 2 and 6 covered by the ledger). This card is skewed to only provide a gossip protocol for orchestration before we jump into a full implementation

Additional context

Adding the diagram that was built from the brainstorming session:

Acceptance criteria

• [ ] We have a state exchange gossip pub/sub between all the nodes
• [ ] The client and the workers follow the protocol defined above with the state exchange pub/sub channel to distribute the work
• [ ] Follow-up card about incorporating #496 and having the ledger select the worker for the client

[mudler]

Blocked by https://github.com/masa-finance/masa-oracle/issues/532

[mudler]

Blocked by https://github.com/masa-finance/masa-oracle/issues/595

[mudler]

Unblocking as we identified that this card doesn't depend on #619