Use sled's approach for testing template executors and consensus logic

[dunnock]

Validator Node is stateful distributed system, which complicates testing as we should account for issues:

• non-critical error states aggregate within a node
• parallel operations interfering (might be impacted by aggregated error state as well)
• distributed / networking failures interfering with operations
• bad mood of node owners, unpaid electricity bills, politics, weather etc

For more background on why this matters read Error Handling in a Correctness-Critical Rust Project. For more inspiration watch "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson

The easiest way to implement such approach I saw is in sled's test_tree_fail_points.rs, where test runner is enumerating possible operations and starts system in controllable environment, then with a property testing feeding random inputs sequences optionally mixed with fail points triggers, validates that every new regular operation output is as expected.

This might be a nice approach for testing:

• Contracts execution logic and error handling for given test smart contracts given random sequence of contracts, each having random input parameters optionally mixed with fail point triggers
• Committee selection, where property testing applied to various response combinations from a VNs from POV of VN, AI
  • We may also cover cases with asset state replication failures
• Coordination in a committee, where property testing applied to various response combinations from a committee nodes from POV of leader, member or observer
  • We may also cover cases with instruction replication failures

[the-mog]

heya, not a bad idea i think. similar to how we are implementing our fuzzer for security testing. the approach: 0) generate initial corpus with proptest 1) derive arbitrary trait for custom types 2) use cargo-fuzz to run the tests.

[dunnock]

@the-mog nice, do you have a link to the actual code?

[the-mog]

not yet.