Query execution observability

[yeya24]

One thing I feel quite limited is the query logger from the original promql engine. It would be great to have observabilities for the queries being executed through the engine and we can see each queries duration, the query plan, detailed stats, etc.

Some more features could be also nice and I am not sure whether it is in scope of this project. Things like slow queries, top queries, etc. That's usually built in in some relational databases and it would be really powerful to collect those query stats like https://docs.pingcap.com/tidb/dev/top-sql.

[GiedriusS]

What do you think about adding a new option for a query such as RecordDuration bool as a starter which would tell the engine to capture the used time by each operator? Then, we could expose it through Explain() :thinking:

[fpetkovski]

I think Explain is meant to be used for printing out the plan, but it should be possible to extend it. We can also surface the information through a different method, like Profile: https://github.com/thanos-community/promql-engine/blob/main/engine/engine.go#L181

[PradyumnaKrishna]

Hello, I would like to work on this as a LFx Mentorship Project.

I have some experience in Golang. I am looking to get more information and resources about Thanos project and this issue, to prepare for LFx Mentorship.

[zeus2611]

hey @yeya24 @GiedriusS @fpetkovski i would like to contribute to this during lfx mentorship period. I have been working on implementing functions in prom-ql engine in thanos. And, i would like to gain more experience and guidance in my open source journey. I'm looking forward to working with you guys.

[GiedriusS]

Please apply on the LFX website 🤗

[saswatamcode]

We have a link for this now: https://mentorship.lfx.linuxfoundation.org/project/a0958ddf-1fd6-4c8e-887f-adb28639a9f4 🙂

[ConnectBhawna]

Hi @GiedriusS @yeya24 @fpetkovski , I am interested in learning about this project and want to work on this project under LFX Mentorship. Please let me know about any materials or resources that I should learn to get a better understanding of the project and its requirements.

Additionally, I was wondering if there is anything else I can do to get started, such as research and learn about the project from the existing documentation.

[GiedriusS]

@ConnectBhawna you can try compiling and running Thanos locally as per instructions on https://thanos.io/tip/thanos/getting-started.md/. Then, you can try switching to this engine with --query.promql-engine=thanos and explore the code in this repository.

[120EE0980]

Helloooo!LFX mentorship brought me here..crossing fingers for great contribution experience.

[J0SAL]

Hello @GiedriusS @yeya24 @fpetkovski, I would love to work on this project. my profile

Please let me know, of any resources or materials I need to know to understand the project better and help me get started.

I have submitted the application for LFX mentorship!

[douglascamata]

FYI according to #thanos-dev slack, @PradyumnaKrishna got the mentorship from LFX to work on this project. Thank you all for the interest 🙏

[GiedriusS]

In my opinion, there are two ways of going around this: embedding observability information into each operator that implements VectorOperator interface (and making ObservableOperator interface that extends VectorOperator and allows extracting o11y info) or passing around some OperatorTracker in each Series and Next call. The latter seems simpler. For example, it allows easily combining certain statistics such as total time taken or memory used. In the former case, we would have to traverse the whole tree again to add up duration or memory used values.

[nishchay-veer]

@saswatamcode Are the applications open for LFX mentorship Term 2 ?

[J0SAL]

Hello, @GiedriusS, @saswatamcode, I am interested in learning about this project and want to work on this project under LFX Mentorship. Please let me know about any good first steps or resources that I should learn to get a better understanding of the project and its requirements. I have submitted my application on the LFX portal

[kanha-gupta]

Hey @yeya24 @GiedriusS I am interested to work on this. I wanted to ask if we provide visualization for every feature in GUI or terminal mode ? Thank you :)