logos-co / nomos-node

Nomos blockchain node
38 stars 13 forks source link

Basic Cryptarchia fuzz test #629

Closed youngjoon-lee closed 1 week ago

youngjoon-lee commented 3 months ago

This contains a basic fuzz test for Cryptarchia. This doesn't cover all possible cases, but I'd like to draw a line here and discuss if this testing strategy is good to move on.

Testing strategy

The Proptest State Machine testing suggests us to mimic the core logic into the ReferenceStateMachine, so that it can be used to generate state transitions and can be compared to the real state machine (aka. SystemUnderTest; SUT) to ensure that the real state machine works as intended. I tried it at the beginning, but found that defining a minimal essential logic from the whole Cryptarchia to be mimicked is not simple. Even if I mimicked a certain logic that I thought essential, the ReferenceStateMachine and SUT didn't work because of missing logic that I didn't mimic. Instead of mimicking, I end up using the real logic for both ReferenceStateMachine and SUT (that we want to test).

In overall, the SUT must always follow the leader chain eventually. However, the follower's chain in the SUT may not grow at this moment, because there is a missing logic (requesting missing parents) in both real code and tests.

NOTE

Unlike Carnot fuzz, I added Cryptarchia fuzz test under the {root}/tests because the entire Cryptarchia is split into multiple crates (engine, ledger, and core). At the beginning, I tried to write fuzz tests for each crate but end up thinking it's not beneficial than I thought.

TODO

We may be able to diverse the testing strategy as below:

youngjoon-lee commented 1 week ago

This PR has been opened for a long time. I don't think this fuzz test is not so beneficial. It would be better to add fuzz tests after having enough unit tests first. Currently, I think this fuzz test is not intuitive and not easy to maintain. I'm closing this PR for now, but please provide your opinion if you have other thoughts.