elastic / rally

Macrobenchmarking framework for Elasticsearch
Apache License 2.0
1.91k stars 314 forks source link

Modify rally to allow multiple cars with complex configuration #1859

Open favilo opened 1 week ago

favilo commented 1 week ago

We'd made some decisions in the past to only allow a single car with a python based configuration.

I'm not certain of the reasoning behind that, but I assume it was because this simplified the evaluation of cars.

This changes that to allow multiple cars to run with complex python based configuration. In particular, it allows us to run APM Tracing and X-Pack-Security simultaneously, which is something I need for benchmarking reasons.

gbanasiak commented 23 hours ago

@favilo Do you think we should add esrally.utils.io and esrally.utils.process to mypy overrides too? My thinking is, if we start adding type hints somewhere, let's mypy do a proper check there, instead of a narrowed-down one. This will raise more mypy errors though, and the scope of PR will increase. Alternatively, we can retract type hint changes from those 2 modules, and add them in a separate PR.

favilo commented 15 hours ago

@favilo Do you think we should add esrally.utils.io and esrally.utils.process to mypy overrides too? My thinking is, if we start adding type hints somewhere, let's mypy do a proper check there, instead of a narrowed-down one. This will raise more mypy errors though, and the scope of PR will increase. Alternatively, we can retract type hint changes from those 2 modules, and add them in a separate PR.

I will add those modules in, I like the idea of decreasing debt continuously. And I still need this branch around for my config to load properly

favilo commented 13 hours ago

@gbanasiak I went a little crazy with those two files... Turned on a stricter mode of mypy than I absolutely needed. But this way, we can be sure we are future proof