elastic / rally

Macrobenchmarking framework for Elasticsearch
Apache License 2.0
1.91k stars 314 forks source link

Update create-track in rally #1836

Closed gareth-ellis closed 1 month ago

gareth-ellis commented 4 months ago

Add option to increase batch size when creating a track to speed up download of data. Update track layout to adhere to best practices

Creating tracks from large corpus can take quite a bit of time, so I have added the option to increase the batch size of scan, so if a user is running with a stable enough network connection and enough hardware resources on the rally instance, they can increase the batch size to speed up download.

I have also updated the track layout to better match how we tend to layout our tracks

ebadyano commented 3 months ago

Thank you so much for the changes! Does it make sense to update command line reference to include the new option to increase the batch size? And update the create -tracks docs?

gareth-ellis commented 3 months ago

Yes, that makes sense. I'll update those too

ebadyano commented 3 months ago

tested with regular index and data streams. I think once we have updated docs it's good to go!

One interesting thing for data-streams scenario: we create a regular index in the new track, it's the same as the original behaviour and out of scope for this pr, but I wonder if for data-streams it makes sense if we create a track that index docs into data streams?..