kv: reconsider performance asserting roachtests

[kvoli]

Is your feature request related to a problem? Please describe. There are roachtests which assert that performance is no less than X, where X is usually a performance number the test gathers before changing the cluster (killing a node, restarting etc).

These tests are sensitive to flakes unrelated to the desired functionality being tested. These tests also reduce a higher fidelity performance number into a binary pass/fail -- losing information.

Describe the solution you'd like Export performance of a run into roachperf and remove any failure conditions on performance.

These tests would be good candidates to start with:

• kv/restart/nodes=12
• rebalance/by-load
• kv/quiescence/nodes=3
• kv/gracefuldraining/nodes=3^1 (see https://github.com/cockroachdb/cockroach/issues/87853)

Describe alternatives you've considered

Performance tracking requires someone(or automated) to periodically check for regressions or improvements and flag them.

When a test fails, the artifacts and debug zip are collected. This won't happen automatically any longer, although we could change this.

We could instead keep the assertion and still collect metrics for viewing performance over time.

Jira issue: CRDB-29517

[blathers-crl[bot]]

cc @cockroachdb/replication