search

HypothesisWorks / hypothesis

Hypothesis is a powerful, flexible, and easy to use library for property-based testing.
https://hypothesis.works
Other
7.53k stars 586 forks source link

Adjust `post_test_case_hook` scope #4022

Closed tybug closed 3 months ago

tybug commented 3 months ago

Move post_test_case_hook higher up so it correctly handles symbolic lifetimes. Reported in https://github.com/pschanely/CrossHair/issues/263, regressed in https://github.com/HypothesisWorks/hypothesis/pull/3977.

Also includes a bandaid for flakiness when replaying interesting non-hypothesis backend examples. I ran into this when we flakily raised DeadlineExceeded with the crosshair backend. I think probably we want to raise Flaky here instead of silently exiting though.

tybug commented 3 months ago

Should we also try to run with crosshair in CI?

Would definitely like to work towards this. I've started taking a look at the remaining blockers here, of which https://github.com/pschanely/hypothesis-crosshair/issues/6 is one.

tybug commented 3 months ago

hold off on this for a sec, the new check doesn't play nicely with symbolic-typed values from crosshair.

tybug commented 3 months ago 
assert type(node.value) is crosshair.libimpl.builtinslib.SymbolicBool

And this is a good reason to be carrying around ir_type separately instead of retrieving it from type(value)!