Closed youngjoon-lee closed 1 week ago
This PR has been opened for a long time. I don't think this fuzz test is not so beneficial. It would be better to add fuzz tests after having enough unit tests first. Currently, I think this fuzz test is not intuitive and not easy to maintain. I'm closing this PR for now, but please provide your opinion if you have other thoughts.
This contains a basic fuzz test for Cryptarchia. This doesn't cover all possible cases, but I'd like to draw a line here and discuss if this testing strategy is good to move on.
Testing strategy
The Proptest State Machine testing suggests us to mimic the core logic into the
ReferenceStateMachine
, so that it can be used to generate state transitions and can be compared to the real state machine (aka.SystemUnderTest; SUT
) to ensure that the real state machine works as intended. I tried it at the beginning, but found that defining a minimal essential logic from the whole Cryptarchia to be mimicked is not simple. Even if I mimicked a certain logic that I thought essential, theReferenceStateMachine
andSUT
didn't work because of missing logic that I didn't mimic. Instead of mimicking, I end up using the real logic for bothReferenceStateMachine
andSUT
(that we want to test).ReferenceStateMachine
is a leader node that proposes new blocks.SUT
(a follower node) in random order.SUT
processes blocks in the order they arrive.In overall, the SUT must always follow the leader chain eventually. However, the follower's chain in the SUT may not grow at this moment, because there is a missing logic (requesting missing parents) in both real code and tests.
NOTE
Unlike Carnot fuzz, I added Cryptarchia fuzz test under the
{root}/tests
because the entire Cryptarchia is split into multiple crates (engine, ledger, and core). At the beginning, I tried to write fuzz tests for each crate but end up thinking it's not beneficial than I thought.TODO
We may be able to diverse the testing strategy as below:
ReferenceStateMachine
, in order to test how the follower handle forks.