search

HypothesisWorks / hypothesis

Hypothesis is a powerful, flexible, and easy to use library for property-based testing.
https://hypothesis.works
Other
7.54k stars 586 forks source link

Use `trace_branch` monitoring for Scrutineer #3781

Open Zac-HD opened 11 months ago

Zac-HD commented 11 months ago

In https://github.com/HypothesisWorks/hypothesis/pull/3776#discussion_r1375259220, we decided to use the trace_line monitoring feature in order to ship a big speedup as soon as possible - and any users on Python 3.12 are now benefiting from that decision 😁

Conceptually though, we should probably use the trace_branch monitor, and return DISABLE to minimize overhead by tracing each branch only once. This should be pretty easy to implement on the tracing side.

I think it might also require some changes to the data structure and display logic to get compatible analyses and reporting; and depending on how that goes maybe also some test changes. For example, I have no idea how it'll handle multi-part conditionals, or ternary statements, etc. when we're not just using physical lines anymore.

https://github.com/HypothesisWorks/hypothesis/issues/3551 discusses some related improvements to Scrutineer's reporting; they can be done in either order but ideally not in parallel.

Zac-HD commented 10 months ago

Note that https://github.com/HypothesisWorks/hypothesis/pull/3797 implies we'll sometimes want to collect line coverage (for observability), branch coverage (for scrutineer), and even at times both. I think this is actually pretty easy to support, and we can add .branches and .lines properties to the tracer to expose the data. And the performance is so much better!