search

racket / htdp

Other
91 stars 69 forks source link

Lack of flexibility in hooks into student testing library #187

Closed dbp closed 1 year ago

dbp commented 1 year ago

Related to #184, I'm trying to (until things get released, ideally more stable than a nightly snapshot), get good property-based-testing support in front of students. All the pieces are there: I can easily use the quickcheck library, and my plan was to provide a #lang that added a few forms to ISL+.

The problem came up when I was trying to add a new check- form: registering the test is fine, but it seems that the reason needs to be one of the predefined ones in https://github.com/racket/htdp/blob/master/htdp-lib/test-engine/test-markup.rkt (specifically, reason->markup will produce #void otherwise, which will fail with a contract error when trying to render the test results). The property-fail? case seems appropriate, but it's defined (specifically, exported with contract) to require the result (what quickcheck spits out) be the struct defined within deinprogramm, not in the quickcheck library.

I can (and have) worked around this by using the deinprogramm version, which then works, but this seems like an unfortunate restriction. It would be nice if I could register tests that had a way of rendering that wasn't already part of the packaged & deployed versions.

mikesperber commented 1 year ago

@dbp Could you clarify what you mean by "not in the quickcheck library"? (The :result record type is defined in deinprogramm/quickcheck.)

mikesperber commented 1 year ago

@dbp So supposedly we could add a case that includes custom markup directly. Would that help? (Would be problematic re translations, but better than nothing.)

I'd rather add appropriate cases to the testing library though if we know them ...

dbp commented 1 year ago

I guess it's a question of whether/to what degree you want clients to be able to plug into the testing framework. I had assumed yes (as that's something I expect of testing frameworks), and parts of the library are exposed (add-test!, etc), so it seems odd that reason has to be one of the builtin ones. Having an escape to provide markup makes sense to me, though I admit to not understanding the library all that well, so take my judgment with a grain of salt.

mikesperber commented 1 year ago

@dbp Absolutely do I want folks to be able to plug into it! (And if I can help elucidate how it works, feel free to push me for docs.)

But before we do an escape - what's the specific case you'd like to add. Or, alternatively, in what way does the current property-fail case not meet your needs?

dbp commented 1 year ago

@mikesperber I didn't actually realize, when I first opened this, that result was exported, sorry! So in this case, I should have been able to adapt the results from the quickcheck library to the deinprogramm/quickcheck version (which is obviously quite similar -- though also, as long as both exist with similar functionality, it's easy for me to just use deinprogramm/quickcheck and avoid that).

One thing that I was surprised by, that maybe is more of a motivation for an escape hatch, is that there isn't a way to take rackunit test suites and adapt them (unless I missed that!). But, perhaps this is a niche use, and at least for me, my use-case is handled at this point, so I'm happy to close this issue.