Add documentation/guide/tooling for query tracing - performance guiding

vespa-engine / vespa

AI + Data, online. https://vespa.ai

https://vespa.ai

Apache License 2.0

5.66k stars 590 forks source link

Add documentation/guide/tooling for query tracing - performance guiding #19670

Open jobergum opened 2 years ago

jobergum commented 2 years ago

Vespa has this awesome feature where you can trace the end to end execution of a query .

This used in combination with timestamp tracing can be used to fully understand the serving performance as it includes end to end tracing of query execution.

This could help guiding on if threads per search can help on e.g latency, cost of first phase versus second phase ranking
Time spent in the stateless container cluster
Analysis of query Blueprint for matching, could fast-search improve matching

So some documentation would be a nice start, a UI tooling component would be even nicer:)

jobergum commented 2 years ago

I'm not sure if we need to treat this as a public API but I have a strong desire to rename the MTF parts to have better explanations before I document this more in-depth? Some inconsistent casing and I'm sure we can use a few bytes more to explain with higher-level abstractions? Thoughts @baldersheim and @bratseth on the API part?

Screenshot 2021-12-13 at 21 46 15

bratseth commented 2 years ago

Agree! Often the best way to see that is to try to write doc for it.

I suggest just creating a list of desired changes, which then can be fixed in one go. It's obvious to us of course that MTF refers to Mordenkainen's Tome of Foes, but it may not be to everybody ... https://en.wikipedia.org/wiki/MTF

jobergum commented 2 years ago

I suggest that we use tag="query_setup" for the events with MTF in the event string, then trace events will all have a tag from the set

query_setup
blueprint => query_execution_plan
match_threads => query_execution

For the MTF strings, I suggest making them more readable?

MTF: Start" => "Read query from network, start query setup"
"MTF: Build query" => "Building query tree"
"MTF: reserve handles" => "..."
"MTF: Fetch Postings" => "Performing dictionary lookups and posting list initialization"
"MTF: Handle Global Filters" => ".."
"MTF: prepareSharedState" => "..":
"MTF: Complete" => "Completed query setup"

jobergum commented 2 years ago

What you think @baldersheim ? I think building tooling around this will be very useful for many

geirst commented 2 years ago

Query tracing was improved in terms of terminology and structure in: https://github.com/vespa-engine/vespa/pull/22217 https://github.com/vespa-engine/vespa/pull/22288

geirst commented 2 years ago

@jobergum The practical search performance guide can now be updated.

baldersheim commented 1 year ago

Soon is in the past now.