search

Zac-HD / hypofuzz

Adaptive fuzzing of Hypothesis tests
https://hypofuzz.com/docs
GNU Affero General Public License v3.0
76 stars 3 forks source link

Fix incompatibility with Pytest 8.x #32

Closed aarchiba closed 6 months ago

aarchiba commented 7 months ago

When I attempt to run hypofuzz on a piece of software with an extensive (mostly non-hypothesis) test suite, it fails. The error message leaves me very unclear what the problem is or what to do about it:

The software is PINT: https://github.com/nanograv/PINT

The hypothesis-based tests include tests of time conversion precision at the nanosecond level for time scales including leap seconds, over time spans of decades. There are also a variety of tests checking support for ill-defined traditional data formats. The test suite has been running successfully using hypothesis for years.

$ hypothesis fuzz -n 4
================================================================================= test session starts =================================================================================
platform linux -- Python 3.11.4, pytest-8.0.0, pluggy-1.4.0
rootdir: /home/peridot/projects/pint/PINT
configfile: pytest.ini
plugins: xdist-3.5.0, dash-2.15.0, hypothesis-6.98.3, anyio-4.2.0, cov-4.1.0
collected 2885 items / 2790 deselected / 95 selected

<Dir PINT>
  <Dir tests>
    <Module test_derived_quantities.py>
      <Function test_companion_mass>
      <Function test_companion_mass_array>
      <Function test_pulsar_mass>
      <Function test_pulsar_mass_array>
      <Function test_omdot_to_mtot>
      <Function test_a1sini_Mc>
      <Function test_a1sini_Mp>
      <Function test_a1sini_Mc_array>
      <Function test_a1sini_Mp_array>
    <Module test_parfile_writing_format.py>
      <Function test_roundtrip>
    <Module test_precision.py>
      <Function test_str_roundtrip_is_exact>
      <Function test_longdouble_str_roundtrip_is_exact>
      <Function test_longdouble2str_same_as_str_and_repr>
      <Function test_time_construction_jds_exact[tai]>
      <Function test_time_construction_jds_exact[tt]>
      <Function test_time_construction_jds_exact[tdb]>
      <Function test_time_construction_mjds_preserved>
      <Function test_time_construction_mjd_versus_jd[tai]>
      <Function test_time_construction_mjd_versus_jd[tt]>
      <Function test_time_construction_mjd_versus_jd[tdb]>
      <Function test_time_to_longdouble_via_jd[tai]>
      <Function test_time_to_longdouble_via_jd[tt]>
      <Function test_time_to_longdouble_via_jd[tdb]>
      <Function test_time_to_longdouble[tai]>
      <Function test_time_to_longdouble[tt]>
      <Function test_time_to_longdouble[tdb]>
      <Function test_time_to_longdouble_utc[mjd]>
      <Function test_time_to_longdouble_utc[pulsar_mjd]>
      <Function test_time_from_longdouble[tai]>
      <Function test_time_from_longdouble[tt]>
      <Function test_time_from_longdouble[tdb]>
      <Function test_time_from_longdouble_utc[mjd]>
      <Function test_time_from_longdouble_utc[pulsar_mjd]>
      <Function test_time_to_longdouble_close_to_time_to_mjd_string[mjd]>
      <Function test_time_to_longdouble_close_to_time_to_mjd_string[pulsar_mjd]>
      <Function test_time_to_longdouble_no_longer_than_time_to_mjd_string>
      <Function test_time_to_mjd_string_versus_longdouble[mjd]>
      <Function test_time_to_mjd_string_versus_longdouble[pulsar_mjd]>
      <Function test_time_to_mjd_string_versus_decimal[mjd]>
      <Function test_time_to_mjd_string_versus_decimal[pulsar_mjd]>
      <Function test_time_from_mjd_string_versus_longdouble_tai>
      <Function test_time_from_mjd_string_versus_longdouble_utc[mjd]>
      <Function test_time_from_mjd_string_versus_longdouble_utc[pulsar_mjd]>
      <Function test_pulsar_mjd_never_differs_too_much_from_mjd_tai>
      <Function test_pulsar_mjd_never_differs_too_much_from_mjd_utc>
      <Function test_time_from_mjd_string_accuracy_vs_longdouble[pulsar_mjd]>
      <Function test_time_from_mjd_string_accuracy_vs_longdouble[mjd]>
      <Function test_time_from_mjd_string_roundtrip[pulsar_mjd]>
      <Function test_time_from_mjd_string_roundtrip[mjd]>
      <Function test_mjd_equals_pulsar_mjd_in_tai>
      <Function test_make_pulsar_mjd_ancient>
      <Function test_make_mjd_ancient>
      <Function test_pulsar_mjd_equals_mjd_on_non_leap_second_days>
      <Function test_pulsar_mjd_equals_mjd_on_leap_second_days>
      <Function test_pulsar_mjd_close_to_mjd_on_leap_second_days>
      <Function test_pulsar_mjd_proceeds_at_normal_rate_on_leap_second_days>
      <Function test_mjd_proceeds_slower_on_leap_second_days>
      <Function test_erfa_conversion_on_leap_sec_days>
      <Function test_erfa_conversion_normal>
      <Function test_d2tf_tf2d_roundtrip[8-1]>
      <Function test_d2tf_tf2d_roundtrip[9-1]>
      <Function test_d2tf_tf2d_roundtrip[10-100]>
      <Function test_d2tf_tf2d_roundtrip[11-1000]>
      <Function test_d2tf_tf2d_roundtrip[12-10000]>
      <Function test_mjd_jd_round_trip>
      <Function test_mjd_jd_pulsar_round_trip>
      <Function test_mjd_jd_pulsar_round_trip_leap_sec_day_edge>
      <Function test_str_to_mjds>
      <Function test_mjds_to_str>
      <Function test_mjds_to_str_roundtrip>
      <Function test_day_frac>
      <Function test_two_sum>
    <Module test_tim_writing.py>
      <Function test_flags>
    <Module test_toa_indexing.py>
      <Function test_select>
      <Function test_getitem_boolean>
      <Function test_getitem_where>
      <Function test_getitem_slice>
    <Module test_toa_reader.py>
      <Function test_numpy_clusterss>
      <Function test_contiguous_on_load>
    <Module test_toa_shuffle.py>
      <Function test_shuffle_toas_residuals_match>
      <Function test_shuffle_toas_chi2_match>
      <Function test_shuffle_toas_clock_corr>
    <Module test_utils.py>
      <Function test_posvel_slice_indexing>
      <Function test_posvel_broadcasts>
      <Function test_posvel_broadcast_retains_quantity>
      <Function test_mjds_to_str_array>
      <Function test_mjds_to_str_array_roundtrip_doesnt_crash>
      <Function test_mjds_to_str_array_roundtrip_close>
      <Function test_str_to_mjds_array>
      <Function test_mjds_to_jds_array>
      <Function test_mjds_to_jds_pulsar_array>
      <Function test_jds_to_mjds_array>
      <Function test_jds_to_mjds_pulsar_array>
      <Function test_compute_hash_detects_changes>
      <Function test_compute_hash_accepts_no_change>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/_pytest/main.py", line 272, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/_pytest/main.py", line 325, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/_pytest/logging.py", line 783, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/_pytest/warnings.py", line 118, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1365, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/_pytest/main.py", line 336, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/_pytest/main.py", line 809, in perform_collect
INTERNALERROR>     hook.pytest_collection_finish(session=self)
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/peridot/projects/pint/venv/lib/python3.11/site-packages/hypofuzz/interface.py", line 37, in pytest_collection_finish
INTERNALERROR>     _, all_autouse, _ = manager.getfixtureclosure(
INTERNALERROR>                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> TypeError: FixtureManager.getfixtureclosure() missing 1 required positional argument: 'ignore_args'

================================================================= 95/2885 tests collected (2790 deselected) in 35.81s =================================================================

Exiting because pytest returned exit code 3
aarchiba commented 7 months ago

I presume I have some forbidden construct in my code somewhere (a combination of hypothesis test and fixture?) but this message gives not the slightest indication where to start finding it.

Zac-HD commented 7 months ago

I just found this independently - it looks like Pytest 8.0 changed the signature of FixtureManager.getfixtureclosure() and so I'll need to add a version-dependent branch. Aiming to get that out in the next day or two 🙂

("it's not you, it's me...")

aarchiba commented 6 months ago

Thanks, works well now!