Started off wanting to understand a bit more on the query set used for our benchmarking, in particular:
If any of queries return non 200 code, if so, which ones are those
If the queries return any docs, if so, are the doc hit count consistent
Which queries are the slowest ones (since we feed in a query file of around 1k entries)
Added some logic in solr-bench and decided to make it a bit more generic as this could be useful in other scenarios too?
Introduced a new detailed-stats flag (default as false) in QueryBenchmark, if enabled, it will keep track of data "per query" in the input query file
Query duration (percentile)
Docs hit count (percentile)
Error count (long)
Since we don't want to make too many changes to the data structure and graphing, for now, I export the detailed stats as the final result json as if it's just another task (task name + metrics type (duration/docs hits/error count) + query executed). ~This should work as is without modifications to the graph generation code~ not true, need some minor change in graph generation/plot code to accommodate data types other than durations.
It probably would create crazy amount of graphs (thousands) in our case, it is mostly useful for us to verify the correctness of the tests, but this will be disabled by default as it might not make that much sense to have such granularity usually 😅
@chatman Can you please take a look and see whether it's useful to have such in master too? Many thanks!
Description
Started off wanting to understand a bit more on the query set used for our benchmarking, in particular:
Added some logic in solr-bench and decided to make it a bit more generic as this could be useful in other scenarios too?
Introduced a new
detailed-stats
flag (default as false) inQueryBenchmark
, if enabled, it will keep track of data "per query" in the input query fileSince we don't want to make too many changes to the data structure and graphing, for now, I export the detailed stats as the final result json as if it's just another task (task name + metrics type (duration/docs hits/error count) + query executed). ~This should work as is without modifications to the graph generation code~ not true, need some minor change in graph generation/plot code to accommodate data types other than durations.
It probably would create crazy amount of graphs (thousands) in our case, it is mostly useful for us to verify the correctness of the tests, but this will be disabled by default as it might not make that much sense to have such granularity usually 😅
@chatman Can you please take a look and see whether it's useful to have such in master too? Many thanks!