Open jobh opened 4 days ago
The consequence of using pytest.warns in internal tests is that the exceptions are never raised, so we don't see the consequences of mis-handling them. Maybe we should always use pytest.raises instead, to ensure the full machinery is excercised.
Ideally, we should check both the warn-and-continue and the raise-warning behaviours. The first part would verify that the warning isn't about anything that leads to another error down the road. See WIP PR #4023 for example
If removing
pytest.warns
here on py3.12,https://github.com/HypothesisWorks/hypothesis/blob/e51d473907120b3c9b441279b83b9cf549c4df80/hypothesis-python/tests/cover/test_monitoring.py#L35
I'd expect a warning to be reported. Instead, an uninformative Flaky exception follows:
(https://github.com/HypothesisWorks/hypothesis/actions/runs/9692069470/job/26744718765?pr=4023)
I guess this can be fixed by f.x. re-raising all warnings as fatal here, https://github.com/HypothesisWorks/hypothesis/blob/e51d473907120b3c9b441279b83b9cf549c4df80/hypothesis-python/src/hypothesis/core.py#L1064
But beyond that: The consequence of using
pytest.warns
in internal tests is that the exceptions are never raised, so we don't see the consequences of mis-handling them. Maybe we should always usepytest.raises
instead, to ensure the full machinery is excercised.