dst: enhancements tracking issue

[asubiotto]

When #883 merges, deterministic simulation testing will be part of our CI pipeline. Currently, deterministic simulation testing is very limited in that it only randomizes scheduling order of goroutines on local run queues. There is more work to do for more interesting failure simulation and infra work.

Simulation work

• [ ] Simulate hard shutdowns: https://github.com/polarsignals/frostdb/pull/895
• [x] Simulate filesystem. It feels like we should be able to do this at the wasip1 syscall layer. This might go hand-in-hand with exploring using the wazero runtime. https://github.com/polarsignals/frostdb/pull/910
• More extensive go runtime simulation
  • [ ] Is there a way to randomize scheduling even more?
  • [ ] Can we improve faketime? Currently time only advances when all goroutines are blocked, causing cases where sleeping goroutines can be starved of CPU. It would be interesting to spuriously wake them up by advancing time in non-conventional scenarios.

Infra work

• [ ] Publish dst.wasm test builds from main. These can be run by anyone with any wasm runtime and GORANDSEED. cc @brancz
• [x] We currently use wasmtime for no good reason apart from it's the default runtime. Experiment with switching to wazero, which also seems to provide more knobs for runtime tuning (e.g. time and randomness). https://github.com/polarsignals/frostdb/pull/896
• [ ] Sometimes runs aren't exactly the same given the same random seed. We need to track this down.