Reinstate coverage

[jobh]

Fixes #4003

Since late March, we haven't had coverage checks active outside of internal.conjecture. We should plug this hole as soon as possible, so it doesn't grow more.

The before-state is of significant size. I will likely just add #pragma: no cover # FIXME to anything that is not immediately obvious, so that we can at least get the configuration change in quickly.

Name                                                      Stmts   Miss Branch BrPart  Cover   Missing
-------------------------------------------------------------------------------------------------------
[elided]
-------------------------------------------------------------------------------------------------------
TOTAL                                                       12917    809   5202     75    93%

This is a draft PR, to mark it as "in progress", but I intend to submit as soon as all checks pass.

Edit: Measurement error, see comment(s).

[jobh]

Update: The big report above shrinks to this when just reverting the pytest-cov/xdist configuration:

Name                                                                                 Stmts   Miss Branch BrPart  Cover   Missing
--------------------------------------------------------------------------------------------------------------------------------
.tox/coverage/lib/python3.10/site-packages/hypothesis/core.py                          666      2    311      1    99%   1107-1108
.tox/coverage/lib/python3.10/site-packages/hypothesis/extra/_patching.py                89      2     29      0    98%   124-125
.tox/coverage/lib/python3.10/site-packages/hypothesis/internal/entropy.py               61      0     18      1    99%   130->145
.tox/coverage/lib/python3.10/site-packages/hypothesis/stateful.py                      454      1    152      0    99%   392
.tox/coverage/lib/python3.10/site-packages/hypothesis/statistics.py                     62      1     43      1    98%   52
.tox/coverage/lib/python3.10/site-packages/hypothesis/strategies/_internal/core.py     793      1    387      1    99%   1821
--------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                12905      7   5208      4    99%

So that's good news in terms of not missing a lot of coverage, but bad news in that the fix is probably less straightforward (worst case reverting parallel execution, but hopefully not).

[jobh]

Aside: I notice that there's one consistently slow test when running under coverage,

============================= slowest 20 durations =============================
102.83s call     hypothesis-python/tests/cover/test_health_checks.py::test_does_not_trigger_health_check_when_most_examples_are_small

We might want to check and possibly reduce it, does it need to loop 100 times? Or alternatively move to nocover. Not in this PR though, as mentioned I think it should not be delayed.

[jobh]

The pandas test failures are all

ValueError: numpy.dtype size changed, may indicate binary incompatibility.
Expected 96 from C header, got 88 from PyObject

which looks like a numpy/pandas binary build thing outside of our control, and I hope they go away if I retry tomorrow :crossed_fingers:

[Zac-HD]

Not sure what's happening with the pandas thing, but the rest looks good to me. Thanks, @jobh!

[jobh]

Not sure what's happening with the pandas thing

It looks like numpy=1.26.4 is installed first, and then numpy=2.0.0 is pulled in along with pandas, leading to a binary conflict with already-built packages.

In the last successful build 3 days ago, this 2.0.0 upgrade didn't happen.

I pushed a temporary fix but maybe it would be better to move the numpy pinned dependency from coverage.txt to test.txt. Are there any test envs that expect numpy to not be installed? Do you have a preference @Zac-HD see below

[jobh]

In the last successful build 3 days ago, this 2.0.0 upgrade didn't happen.

...indeed, yesterday was the release date of numpy 2.0.0 (https://numpy.org/news/). And the pandas wheel doesn't build with numpy=1.26.4 (version skew, possibly related to build isolation).

I need advice to continue. I guess the simplest would be to upgrade our main numpy dep to 2.0.0 and perhaps later add a runner for testing the 1.26 branch.

(edit: Maybe it will sort itself out if I just wait for the next update-pinned-dependencies? I got it to green now by manually pinning numpy in tox.ini and updating the pandas test matrix for availability of binary wheels. But the manual pinning will have to be fixed/removed.)

[jobh]

@Zac-HD Could you indicate how you'd like to proceed, whether we should

1. wait for update-pinned-deps and rebase on top of that, or
2. merge as-is and deal with the fallout when doing update-pinned.

Possibly I could trigger the update myself, but I don't know if there's some magic sauce to be added.

Note, regardless of (1) or (2) we should disable the pandas11/12 check requirement as there doesn't seem to be binary packages available for those on py3.10.

[jobh]

Thanks @Zac-HD! Everything seems to be nicely unblocked now :+1:

I'll merge it as-is, and ensure the issue stays open; I'd prefer to take on the FIXMEs separately.