apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.51k stars 1.29k forks source link

[multi stage] Measure threadCpuTime metric (similar to existing engine) in the operators, aggregate and include in response #9414

Open siddharthteotia opened 2 years ago

siddharthteotia commented 2 years ago

In the current engine, we measure threadCpuTime metric at the BaseCombineOperator level. Aggregated metric is sent to servers and broker aggregates across servers and includes in query response stats. We need to do something similar (granularity etc to be discussed) in the multi-stage engine.

walterddr commented 2 years ago

good idea. we also had some feedback from the community on this. but the stats definition is the first to sort out.

siddharthteotia commented 2 years ago

I agree. Measurement is probably ok but correct propagation, aggregation and reflecting at different stages is also something that needs to be thought about