scipy / scipy_doctest

Floating-point aware doctesting
BSD 3-Clause "New" or "Revised" License
5 stars 4 forks source link

Replace `nameerror_after_exception` by pytests's `continue_on_failure` #88

Open Sheila-nk opened 1 year ago

Sheila-nk commented 1 year ago

https://github.com/ev-br/scpdt/pull/87#issuecomment-1677434321 please make sure that this test passes with the plugin

_Originally posted by @ev-br in https://github.com/ev-br/scpdt/pull/87#discussion_r1293579963_

ev-br commented 6 months ago

This came first in https://github.com/scipy/scipy/pull/13116 , might help cooking up a standalone test case. Once there is a test, it'll be easier to decide if this can be merged with --continue_on_failure in the plugin layer.

ev-br commented 6 months ago

A test exists for the doctest layer in test_testmod.py::TestNameErrorAfterException. The CLI incantation is

$ pytest scpdt/tests/failure_cases_2.py --doctest-modules --doctest-continue-on-failure

and it indeed seems to match: with --doctest-continue-on-failure the traceback has four exceptions (as expected for name_error_after_exception=True); without it we only get one.

So need to golf sync the DTConfig setting and the pytest setting:

ev-br commented 6 months ago

Looked at it a while. This is low prio TBH.

A worthу aim would be to remove PytestDTRunner.report_unexpected_exception and .report_failure which were vendored from pytest's runner, https://github.com/pytest-dev/pytest/blob/8.0.2/src/_pytest/doctest.py#L184

However, this would require to multiple inherit PytestDTRunner from both DebugDTRunner (which it does now) and pytest's PytestDoctestRunner.

Does not seem worth it TBH.

Also, the PytestDTRunner inherits from DebugDTRunner, so it stops after the first exception anyway. Also, DebugDTRunner overrides all logic from nameerror_after_exeption.

ev-br commented 2 weeks ago

From the code health perspective, it's probably best to simply remove the nameerror_after_exception and always use the pytest naming, continue_on_failure throughout, both in pytest and "basic" doctest layer.