opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.68k stars 1.79k forks source link

Update Segment replication stats APIs to support pull based architecture #15534

Open mch2 opened 1 month ago

mch2 commented 1 month ago

With #4577 replicas will sync directly with their source of replication rather than pushing updates from the primary.

Today stats are collected at the primary level to support APIs and enforce backpressure. With the rw split we will ensure primaries do not have any direct communication with replicas and will not be able to collect these stats.

To fix this we can update our stats APIs to fetch the latest checkpoint from each replica directly and compute the required stats at the coordinator, eliminating the need for primaries to capture these stats.

vinaykpud commented 2 weeks ago

Hi @mch2 can I take this task?