nickelization
commented
1 year ago Hi @hamiltop! That is a good question, and there is definitely some overlap between the two projects. I haven't had time to dig in too deeply to proptest-state-machine so I may be missing some key differences, but currently the biggest one is probably that proptest-stateful is geared more toward testing async APIs. (It can certainly be used to test non-async code, but it depends on tokio and uses async-trait to specify async trait methods, and also includes extra callbacks for test case setup/teardown.)
On the flip side, the proptest-state-machine library includes the ability to attempt shrinking individual operations themselves after it finishes the initial shrinking phase of removing entire operations. But I'm not sure how useful this capability would be in practice, because in most stateful property tests I've written, it's pretty likely that any shrinking attempt on an individual op will cause a precondition to be violated, which means the actual opportunities for shrinking in this way are often pretty rare. I've found that only removing entire ops generally gives good enough results in practice, anyway. We have thought about trying to add this as an enhancement in a way that works with preconditions, though, and there's more on this topic in issue #5.
When we first started working on proptest-stateful I actually checked a bunch of places (including the proptest repo) to see if anything like this existed or was being planned, and I didn't find anything about proptest-state-machine or any other similar libraries. If it had existed last year, we might have built off of that instead, although again, I haven't evaluated it that closely, so I don't know what kind of features it might be missing that we would need.
I also originally thought about trying to merge this code upstream to proptest early on, but thought it made more sense to keep it as a separate crate. However, maybe that thinking was misguided since now there is similar code that's been integrated into the main proptest repo 😄. At some point I may look into merging functionality from proptest-stateful upstream, but we would need to rewrite all the existing test suites we've already written at ReadySet if we wanted to deprecate use of our own library, so it's probably not a high priority project for us. Hopefully in the short term the community will benefit from having both options available!
matthew-russo
Cool project! Captures a pattern we've used internally and makes things a little more clear.
How does this differ from https://github.com/proptest-rs/proptest/tree/master/proptest-state-machine and would it make sense to try and unify them? Or maybe they are separate but it still might make sense to pull this functionality into the main proptest repo and docs.