search

redis / redis-benchmarks-specification

The Redis benchmarks specification describes the cross-language/tools requirements and expectations to foster performance and observability standards around redis related technologies. Members from both industry and academia, including organizations and individuals are encouraged to contribute.
Apache License 2.0
30 stars 11 forks source link

[BUG]: Failures in benchmarking tests that use dataset configured in dbconfig #242

Open odidev opened 1 year ago

odidev commented 1 year ago

Describe the bug

I am executing benchmarking tests on the Linux/AMD64 AWS instance.

I am facing issues in all the tests that have dataset configured in dbconfig. For example, consider the test “memtier_benchmark-1key-geo-60M-elements-geohash-pipeline-10.yml”. I ran the test with dataset, following the command below: 

redis-benchmarks-spec-client-runner --db_server_host localhost --db_server_port 6379 --client_aggregated_results_folder ./test --flushall_on_every_test_start --run-tests-with-dataset –test memtier_benchmark-1key-geo-60M-elements-geohash-pipeline-10.yml

The failure logs are as below: 

2023-06-27 10:43:03 ERROR The total numbers of keys in setup does not match the expected spec: 1!=0. Aborting... 
2023-06-27 10:43:03 CRITICAL Some unexpected exception was caught during local work. Failing test....

I found a similar issue here: < https://github.com/redis/redis-benchmarks-specification/issues/206 >, and the solution was this PR: < https://github.com/redis/redis-benchmarks-specification/pull/213/files >.

Following the above PR, I edited the “keyspacelen” from 1 to 0, and the test passes successfully.

Similarly, there are 8 more tests in the suite, that use dataset. Below is the list:

memtier_benchmark-1key-geo-60M-elements-geohash.yml memtier_benchmark-1key-geo-60M-elements-geopos-pipeline-10.yml memtier_benchmark-1key-geo-60M-elements-geopos.yml memtier_benchmark-1key-geo-60M-elements-geosearch-fromlonlat-bybox.yml memtier_benchmark-1key-geo-60M-elements-geodist-pipeline-10.yml
memtier_benchmark-1key-geo-60M-elements-geosearch-fromlonlat-pipeline-10.yml memtier_benchmark-1key-geo-60M-elements-geodist.yml
memtier_benchmark-1key-geo-60M-elements-geosearch-fromlonlat.yml

All the above-mentioned 9 tests use dataset and passes successfully after editing “keyspacelen” from 1 to 0.

Do you think it is reasonable to edit the “keyspacelen” to 0? If yes, I will raise a PR for the same.

Kindly let me know your thoughts on the above.

Environment (please complete the following information):

odidev commented 1 year ago

Hi team, a gentle reminder!! Can you please look into this issue?

odidev commented 1 year ago

Hi Team, a gentle reminder!! Can you please look into this ticket?