search

jqwik-team / jqwik

Property-Based Testing on the JUnit Platform
http://jqwik.net
Eclipse Public License 2.0
578 stars 64 forks source link

Programmatic use of jqwik without JUnit #503

Closed FeldrinH closed 2 months ago

FeldrinH commented 1 year ago

Testing Problem

Jqwik seems to have a comprehensive and well maintained set of core algorithms for property based testing (generators, shrinking etc). Sometimes it would be useful to use them but for whatever reason you can't use JUnit Platform to run jqwik tests.

Suggested Solution

It would be nice if there was some public method that would allow you to simply supply the predicate as a function, arbitraries for the parameters, and some configuration options and it would run the testing algorithm and return a shrunk set of arguments that violates the predicate (if they are found). Something similar to the API of the currently unmaintained QuickTheories library.

Discussion

Arbitraries seem to already have a public API, but the shrinking logic seems to be at least partially private. Exposing more of the shrinking machinery seems like it might need a pretty sizable refactor (from a quick look at the source code it seems that the shrinking algorithm related classes are quite tightly coupled to a lot of internal classes which are in turn coupled to test engine classes).

Overall I don't see any long-term disadvantages, just that it might be a lot of work to extract the relevant parts from internal classes.

jlink commented 1 year ago

It's fair to discuss jqwik's programming model. I prefer the current one over the QuickTheories approach because it allows me to leverage the JUnit Platform's way of test discovery, execution and reporting. Some of that I couldn't do otherwise.

That said, it wouldn't be too much effort to expose some parts of the internal API to use it in an independent way. I have it in the back of my mind for jqwik2, which will hopefully see its first beams of sunlight as early as this year. So stay tuned.

FeldrinH commented 1 year ago

It's fair to discuss jqwik's programming model. I prefer the current one over the QuickTheories approach because it allows me to leverage the JUnit Platform's way of test discovery, execution and reporting. Some of that I couldn't do otherwise.

That's fair, I can see what advantages leveraging JUnit infrastructure has here. That said I personally definitely prefer the QuickTheories approach. Partly because it's more flexibile (you can use it in any test engine or no test engine at all) and partly because it's more lightweight (you just don't have to bring in the entire JUnit infrastructucture just to check a single property).

As far as I know there aren't any libraries similar to QuickTheories for Java that are actively maintained, so I will remain cautiously optimistic that jqwik2 can fill at least some of that void.

jlink commented 1 year ago

That said I personally definitely prefer the QuickTheories approach. Partly because it's more flexibile (you can use it in any test engine or no test engine at all) and partly because it's more lightweight (you just don't have to bring in the entire JUnit infrastructucture just to check a single property).

Absolutely. Drawback: Less powerful

As far as I know there aren't any libraries similar to QuickTheories for Java that are actively maintained, so I will remain cautiously optimistic that jqwik2 can fill at least some of that void.

If my main goal was to increase market share, I'd go for a lightweight API plus Jupiter extension. Sadly enough, I mostly do it out of joy and personal challenge ;-/

jlink commented 2 months ago

It won't happen within jqwik 1. But it's one of the core points of jqwik 2. See https://github.com/jqwik-team/jqwik2-api-discussion