masa-finance / masa-oracle

Masa Oracle: Decentralized Data Protocol 🌐
https://developers.masa.ai/docs/category/oracle-node
MIT License
16 stars 15 forks source link

feat(workers): Implement adaptive worker selection for improved task distribution #544

Open teslashibe opened 3 weeks ago

teslashibe commented 3 weeks ago

Problem:

The current worker selection process is inefficient, often failing to utilize all available workers and potentially leading to increased latency and uneven work distribution.

Acceptance Criteria:

  1. Implement a scoring system for worker reliability and performance.
  2. Create a cache of recently successful workers.
  3. Develop a selection algorithm that chooses a subset of high-scoring, available workers.
  4. Implement fallback mechanism to try additional workers if initial attempts fail.
  5. Ensure the system adapts by updating worker scores based on performance.
  6. Maintain or improve current task completion success rate.
  7. Reduce average latency for task completion by at least 20%.
  8. Handle at least 30% more concurrent tasks without degradation in performance.
  9. Add logging for worker selection process to aid in debugging and optimization.
  10. Include unit tests for new components and integration tests for the overall worker selection process.

Out of Scope:

teslashibe commented 2 weeks ago

@Luka-Loncar add this to 0.8.1 please

teslashibe commented 2 weeks ago

Do we kill this in favor of https://github.com/masa-finance/masa-oracle/issues/507 or do we merge the two to become a hybird?