danielmitterdorfer
commented
5 years ago A few times I accidentally had an error in the search operation definition, for example I misspelled the name of the field or structured the query incorrectly. Both the contender and baseline were returning no results, but I simply thought my optimization wasn't working.
By default Rally will just continue on errors and report an error rate at the end of the benchmark but I think you can already achieve what you're after by specifying --on-error=abort? It would abort the benchmark as soon as Elasticsearch returns an error and provides more context (i.e. the response body from Elasticsearch). Would that help you?
I wonder if it'd be useful for rally to return the number of hits from a search operation, or even the whole search response
Rally already returns the number of hits in the metrics metadata but you need to configure a (dedicated) Elasticsearch metrics store for that. I think it would cover most of your needs. Here is an example from our nightly benchmarks:
{
"environment": "nightly",
"race-id": "6cac8c34-93b4-416e-a1ea-6c4e1b87f9fd",
"race-timestamp": "20191005T200042Z",
"@timestamp": 1570306375355,
"relative-time": 41959,
"track": "geonames",
"challenge": "append-no-conflicts",
"car": "defaults",
"operation": "phrase",
"task": "phrase",
"name": "service_time",
"operation-type": "Search",
"sample-type": "warmup",
"value": 3.7241727113723755,
"unit": "ms",
"meta": {
"success": true,
"distribution_flavor": "oss",
"source_revision": "d598083051d46e7d44f577473dde52c4a32f6776",
"distribution_version": "8.0.0-SNAPSHOT",
"tag_license": "oss",
"timed_out": false,
"took": 1,
"hits": 58,
"hits_relation": "eq"
}
}In the meta.hits property we already track the number of hits and the meta property probably contains a few more things you're interested in.
jtibshirani
talevy
gingerwizard
As part of developing a new search feature or optimization, I often run benchmarks by adding a new search operation to an existing rally track. I've found that it can be easy to make mistakes in setting up the benchmark:
For this use case, I wonder if it'd be useful for rally to return the number of hits from a search operation, or even the whole search response. Seeing that the baseline and contender return the same number of results (and return more than 0 results) would give confidence that I've set up the benchmark correctly and help avoid some manual testing.