search

jqwik-team / jqwik

Property-Based Testing on the JUnit Platform
http://jqwik.net
Eclipse Public License 2.0
575 stars 64 forks source link

Support live visualizations with Tyche, via the `OpenPBTStats` log format? #604

Open Zac-HD opened 3 days ago

Zac-HD commented 3 days ago

Tyche is a VS Code extension designed to help users understand what their property-based tests are doing - with nice features like live-updating graphs, and lots of explorable raw data if you want to dig deeper. I've been collaborating with the author since late last year (as a Hypothesis maintainer), as described in our paper. Notably, this helped us find several performance problems and outright bugs in Hypothesis itself, as well as being useful for our users.

But the reason I'm opening this issue is that Tyche is not Hypothesis-specific; it's instead based on a simple json-lines format which could be emitted from any PBT library and consumed by a wide variety of tools. If you want to support it from jqwik, emitting the type, run_start, property, status, and representation is enough to enable a substantial portion of Tyche’s features.

jlink commented 3 days ago

@Zac-HD Thats definitely interesting. I'll have a look at the format. Does the VS Code integration require anything else than emitting json lines?

Zac-HD commented 3 days ago

Nope - the Tyche editor plugin watches known directories for the jsonlines file, and everything from there is automatic. Users can configure which directory that is, but @hgoldstein95 tends to add support for new locations so they mostly won't even need to do that 😁

jlink commented 7 hours ago

@Zac-HD Is there a showcase project where I could see how Tyche looks with hypothesis? I need something cool to motivate me :-)

Zac-HD commented 7 hours ago

image

and of course you can click in to any of these slices to see each test-case by category; load the data up into datasette or pandas and explore it that way, etc.

I don't have a demo project, the whole point is that it works with any test suite, but https://github.com/python-jsonschema/hypothesis-jsonschema is a nice size to explore and has many property-based tests if you want to try it out?