sunithabeeram
commented
5 years ago https://github.com/apache/incubator-pinot/pull/3942 tracks Completeness (a) part of the issue. Note the following caveats:
- There is a legitimate case where the servers may not have segments available - this happens during retention. We don't consider to be a serving error. The change makes a best effort attempt to track recently deleted segments using an in-memory cache. This does mean that on server restarts the segments retained out segments can be marked as missing.
- When segments move or get refreshed with explicit state transitions (ie, an online->offline transition is triggered followed by a offline->online transition), and the query reaches a server after the online->offline transition), the server won't have the segment and we will count them as partial responses.
This issue tracks the need to expose better metrics in Pinot to expose completeness and freshness in the query path. Today, a lot of individual metrics are available for tracking, but its typically hard to say which queries were actually affected when issues do happen. Allowing tracking/metadata to be sent in query response helps with this. This issue tracks efforts being made for this. NOTE: the tracking is on a best effort basis - we would like to track the most common possibilities (without tying ourselves to specific individual scenarios) without compromising the execution time of the query and continuing to keep the design/implementation as simple/intuitive as possible.
Details:
Completeness:
Currently, we set a "partial-response" metadata flag when a broker is unable to get a connection to the server or response from a server within configured timeout. We would like to expand this to cover: (a) cases where servers don't have the segments the broker is requesting. (b) broker's routing table is incomplete - for ex, when all the replicas of a given segment are ofline or in error state.
Freshness:
In the current scope, we limit the freshness of the data to realtime tables. We would like to measure how "recent" the data available for querying in realtime is.
More details for each of these will follow.