search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.52k stars 1.29k forks source link

RefreshSegmentTaskGenerator: Evaluate and Use Server APIs to determine if refresh is needed. #14483

Open vvivekiyer opened 4 days ago

vvivekiyer commented 4 days ago

PR https://github.com/apache/pinot/pull/14300 talks about the reasoning for triggering no-op minion tasks when there are table config updates.

Cons of this approach is that there will be minion tasks created for all segments for each table config update.

To overcome this problem, we can use a server side API that will return the list of segments to be refreshed. It is being developed in https://github.com/apache/pinot/issues/14450.