db: better testing of error handling pathways

[jbowens]

cockroachdb/cockroach#63847 is likely an example of improper error handling introducing corruption. As a part of the quality initiative, we should make an effort to audit and improve testing of error handling code paths.

We currently have some existing tests like TestIngestError that try injecting an error at every i-th operation. There are probably more code paths that deserve this testing treatment. That particular test, TestIngestError, might also benefit from more assertions around the state of the resulting DB.

The metamorphic test has an -error-rate that will inject errors into read-only operations and automatically retry them, but it currently defaults off. We can update it to default to some low rate.

Related to #270.

Jira issue: PEBBLE-212

[jbowens]

The metamorphic test has an -error-rate that will inject errors into read-only operations and automatically retry them, but it currently defaults off. We can update it to default to some low rate.

^ This item is a quick win, so triaging as such. Can move to backlog afterwards.

[jbowens]

Remaining work:

• We still need better integration with the metamorphic tests: The metamorphic tests should retry operations that fail with ErrInjected, but retrying relative positioning operations is a bit tricky.
• We don't have randomized testing of all write paths too. This has its own challenges since so many write paths are fatal if they error.