search

fullstorydev / solr-bench

Solr benchmarking and load testing harness
Apache License 2.0
17 stars 10 forks source link

Allow detailed-stats on query benchmark (old branch) #59

Closed patsonluk closed 1 year ago

patsonluk commented 1 year ago

Description

Started off wanting to understand a bit more on the query set used for our benchmarking, in particular:

  1. If any of queries return non 200 code, if so, which ones are those
  2. If the queries return any docs, if so, are the doc hit count consistent
  3. Which queries are the slowest ones (since we feed in a query file of around 1k entries)

Added some logic in solr-bench and decided to make it a bit more generic as this could be useful in other scenarios too?

Introduced a new detailed-stats flag (default as false) in QueryBenchmark, if enabled, it will keep track of data "per query" in the input query file

  1. Query duration (percentile)
  2. Docs hit count (percentile)
  3. Error count (long)

Since we don't want to make too many changes to the data structure and graphing, for now, I export the detailed stats as the final result json as if it's just another task (task name + metrics type (duration/docs hits/error count) + query executed). ~This should work as is without modifications to the graph generation code~ not true, need some minor change in graph generation/plot code to accommodate data types other than durations.

It probably would create crazy amount of graphs (thousands) in our case, it is mostly useful for us to verify the correctness of the tests, but this will be disabled by default as it might not make that much sense to have such granularity usually 😅

@chatman Can you please take a look and see whether it's useful to have such in master too? Many thanks!

chatman commented 1 year ago

Sounds good, Patson. Reasonable to keep it disabled by default. +1 to merge.

patsonluk commented 1 year ago

Closing this in favor of #61