[Roadmap] System verification for Xline

[bsbds]

Description:

Background:

The purpose is to create a robust system verification process for Xline. It allows us to deterministically capture errors in development, and finally validate that the system is working as intended.

Objectives：

The system verification for Xline roughly consist of two parts

• Deterministic simulation

  The deterministic simulation will eliminate side effects of the system.

• Testing framework integration

  The testing framework will allow us to do chaos engineering for the system. While the testing framework may not run the system deterministicly, the deterministic simulation is needed to be done first.

Roadmap:

This is a preliminary outline of the roadmap. It is subject to further refinement and additional details may be added in the future.

1. Deterministic simulation

• [x] #357

  Firstly we would introduce the simulator to refactor the system in a more elegant way. For instance we could utilizing the simulator to remove the inject network filter in integration tests.

• [ ] Full deterministic simulation support for the entire system

  Our goal is be able to mock all IO interfaces and other non-deterministic sources in Xline, so that side effects of the system would be eliminated. This can be achieved by integrating Madsim into other parts of the system.

  However, it is important to note that the Madsim project is still in unfinished state(such as the file system simulation), we may need to do some contribution to the project.

2. Testing framework integration

• [ ] Set up testing framework for chaos engineering

  We may introduce Jepsen for system testing. This would involve setting up the testing framework specificly for Xline, writing tests and introducing failures.

• [ ] Integration with Madsim

  To deterministicly reproduce the issues found by tests, it will be necessary to integrate the Jepsen with Madsim. Which makes it more easy to identified the issue.

[bsbds]

This doc is outdated and will be closed