czan / stateful-check

Stateful generative testing in clojure
MIT License
117 stars 11 forks source link

Results protocol #15

Closed r0man closed 1 year ago

r0man commented 1 year ago

Hi @czan,

I thought before designing the data structures for the debugger I first take a look what has changed in test.check recently with regards to error reporting, and here is what I discovered.

Newer versions of Clojure test.check have a Results protocol [1]. What this protocol allows is to return additional data from a property, instead of just true or false. If we are using this protocol, we could get rid of the stateful-check failure exception machinery.

This has the benefit of having more readable test.check reports returned by run-specification. The values under the :result key and the :shrunk :smallest keys are no longer exceptions (with a long stacktrace attached to them, that makes it hard to read), but the plain result-data instead, which is basically the ex-data of the failure expression.

I think with this change, and some further tweaks to the error reporting, we could leaverage more of CIDER and Clojure Test error reporting functionality. I think we should look more closely to how clojure.test.check.clojure-test is reporing, which I plan to do in maybe another PR?

Wdyt?

czan commented 1 year ago

Hi @r0man!

I took a look at this PR, and I ended up implementing the same thing with some different code in 9040bda8a97a3a8d130cd481c4e3bbe0c761de4d. I've put that change up on the develop branch, along with pulling in the first commit in this PR (the test.check version bump).

Thanks for the ideas and patches. I think we can close this PR now. :slightly_smiling_face:

r0man commented 1 year ago

Nice. Thank you!