kvserver: sep-raft-log: add crash-testing datadriven test harness

[tbg]

We need to exhaustively test the invariants for the engine states in separate raft logs. We'll add assertions but should also explore these using datadriven tests.

Some of the code in https://github.com/cockroachdb/cockroach/pull/88606 might be helpful.

TODO:

• [ ] crash in the middle of snapshot ingestion
• [ ] crash in the middle of replica subsumption (snapshot and merge trigger case)
• [ ] crash in the middle of replica creation
• [ ] crash in the middle of split
• [ ] crash in the middle of merge
• [ ] crash in the middle of replica destruction
• [ ] general engine-level testing of all of the above.

I'll add that our testing of the snapshot SSTs is currently weak. We rely on spinning up a real cluster and then intercept the SSTs, which is not very maintainable. We should complement this with kvstorage-level tests that create a snapshot and examine in detail their contents and the engine each one targets.

Epic: CRDB-220

Jira issue: CRDB-22240

[blathers-crl[bot]]

cc @cockroachdb/replication

[tbg]

We have a basic datadriven framework at https://github.com/cockroachdb/cockroach/blob/672e8b1e628ef0a4c920cf42355c44cbc7a625b6/pkg/kv/kvserver/kvstorage/datadriven_test.go that should, with extra work, be able to accommodate our testing needs.

The testing it is used for is limited. I'm adding some bullets to the comment above about missing areas.