As of #76582, we now use multi-tenant test setup in 50% of the tests, which includes the benchmarks. We probably forgot to consider the impact of that change on running the benchmarks, and there are a couple of things we need to address (quoting Raphael):
one is the cosmetic issue of the info messages (when the multi-tenant setup is used), causing the benchstat utility to get confused. This can be reliably addressed in multiple ways, by tweaking the messages (e.g. adding a newline character at the beginning).
the other is the question of non-determinism in benchmark outputs (TBD whether this non-determinism actually takes place). Here the situation is that we have automation that tracks benchmark regressions over time. This automation is currently not aware of the multi-tenancy randomization. We need to enhance that.
Separately, we need to make peace with the idea that single tenancy is going away. At some point in the future, we will transition all our bench tests to run on a multi-tenant cluster, and we are expecting no performance regression. If we are seeing a performance regression today, this means that there is multi-tenancy optimization work to do that needs to be planned and done by the appropriate teams.
As a quick fix, #83254 disabled the usage of the multi-tenant setup under bench.
As of #76582, we now use multi-tenant test setup in 50% of the tests, which includes the benchmarks. We probably forgot to consider the impact of that change on running the benchmarks, and there are a couple of things we need to address (quoting Raphael):
benchstat
utility to get confused. This can be reliably addressed in multiple ways, by tweaking the messages (e.g. adding a newline character at the beginning).Separately, we need to make peace with the idea that single tenancy is going away. At some point in the future, we will transition all our bench tests to run on a multi-tenant cluster, and we are expecting no performance regression. If we are seeing a performance regression today, this means that there is multi-tenancy optimization work to do that needs to be planned and done by the appropriate teams.
As a quick fix, #83254 disabled the usage of the multi-tenant setup under bench.
Jira issue: CRDB-17081
Epic CRDB-31933