Tracking: deterministic simulation testing

[wangrunji0408]

We are utilizing deterministic simulation testing to find bugs efficiently and reproduce them at any time.

Status and TODO

• [x] Unit test
  • [x] Find out why some tests failed
  • [ ] Simulate fs API for disk object store
  • [x] #6672
• [x] End-to-end test
  • [x] Batch and Streaming
  • [x] Cluster: 3 compute + 1 frontend
  • [x] Make things deterministic
  • [x] Compactor
  • [x] #5181
  • [x] Parallel test @KveinAxel
  • [x] Etcd
  • [x] #5161
  • [x] #5170
  • [x] #12707
• [ ] Fault injection
  • [x] #4527
  • [x] #5010
  • [x] #6424
  • [x] #5655
• [ ] Other tools
  • [x] #6665

Pull Requests

• 2477

• 2718

• 2787

• 2888

• 3951

• https://github.com/madsys-dev/madsim/pull/16

• 4176

• 4336

• 4460

• 4476

• 4550

• 4675

• 4738

• 4843

• 4917

• 4950

• 5084

• 5342

• 5536

• 5657

• 6676

• 6708

• 6778

Hunted Bugs

• type: execution order introduced by #4437 fixed by #4516

• 4875

• 4782

• 5143

• 5103

Related Links

• "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson
• sled simulation guide (jepsen-proof engineering)
• Testing sync at Dropbox
• BUGGIFY
• Project Coyote from Microsoft
• Project Hermit from Facebook

[lmatz]

Is it possible to add e2e simulation tests for the connector sink or source? e.g. https://github.com/risingwavelabs/risingwave/pull/6631

[wangrunji0408]

Is it possible to add e2e simulation tests for the connector sink or source? e.g. #6631

Theoretically it's possible. But we have to build a dedicated simulator for each kind of source and sink. Each of them will take us at least 0.5 man-months in my prediction. Currently we already have a simulator for kafka. With this, the complete data flow from source to sink has been covered in the test. So any more simulator for a connector will only result in the connector code itself being tested, which I don't think the ROI is high. 🤔

By the way, I just found a new project hermit published by Meta a week ago. It is a system-level deterministic testing tool. It seems like a good solution to this case. I'm going to do some research on it soon. (UPDATE: tracking in #6665)