Exclude tasks based on serverless status

[pquentin]

When running Rally benchmarks on Elasticsearch Serverless, some operations do not make sense depending on the operator status, such as force merge. To make running benchmarks simpler, this pull requests skips some operations automatically based on the operator status of the user.

I've tested this with the PMC track that has a challenge with a parallel operation by explicitly setting serverless.operator to False:

esrally race --track-path=$HOME/src/rally-tracks/pmc --test-mode --target-hosts=... --pipeline=benchmark-only --client-options=serverless.json --user-tags="intention:rally-serverless" --on-error=abort --challenge=indexing-querying

It prints the following at the beginning of the race:

[INFO] Race id is [71666d4e-121b-47a8-b7c2-8b2550434e95]
[INFO] Treating parallel task in challenge [indexing-querying] as public.
[INFO] Excluding [put-settings], [check-cluster-health], [force-merge], [wait-until-merges-finish] as challenge [indexing-querying] is run on serverless.
[INFO] Racing on track [pmc], challenge [indexing-querying] and car ['external'] with version [8.10.0].

TODO:

• [x] Fix rally-tracks-compat
• [x] Add escape hatch

Sub commands to test:

• [x] race
• [x] list
• [x] delete
• [x] info
• [x] create-track
• [x] compare
• [x] build
• [x] download
• [x] install
• [x] start
• [x] stop
• [x] add

[gbanasiak]

Forgot to mention I also went through all operation types and compared with serverless protection status as of July 27th and found no discrepancies.

[pquentin]

You do need to manually set serverless.operator to False in the source code. This will be detected in a later pull request.

[dliappis]

No but you need to manually set serverless.operator to False, sorry. This will be detected in a later pull request.

Where, in rally.ini under driver? If yes, I did that, but couldn't get the behavior you described i.e. [INFO] Excluding [put-settings], [check-cluster-health], [force-merge], [wait-until-merges-finish] as challenge [indexing-querying] is run on serverless.

[dliappis]

You do need to manually set serverless.operator to False in the source code. This will be detected in a later pull request.

Ok after explicitly setting False in https://github.com/elastic/rally/pull/1760/files#diff-9cd59b2fffb801ee46e1819d3dc79a407b1b5784a09d535b3ff68820f9c4102dR204, I managed to get the expected output

[INFO] Treating parallel task in challenge [indexing-querying] as public.
[INFO] Excluding [put-settings], [check-cluster-health], [force-merge], [wait-until-merges-finish] as challenge [indexing-querying] is run on serverless.

It's unclear to me why explicitly setting that value in rally.ini under [driver] wouldn't achieve the same result though.

[pquentin]

I fixed reading from rally.ini. Please take another look!