Feature: Post-mortem mode for exceptions or failed test cases with DAP

[wookayin]

Neotest is aware of DAP, e.g., neotest.run.run({strategy = dap}) would run test cases in a DAP debug session.

One missing feature is that when an exception happens or a test fails (via AssertionError), the DAP sesion does not hit the breakpoint and the program terminates normally. Think of some behaviors like pytest --pudb, which would launch the PuDB debugger session with the breakpoint hit on the failing line (like a post-mortem mode).

It would be nice we can have a similar breakpoint or post-mortem mode with the DAP session when running via dap strategy. The current behavior is just showing the test output in a floating window, not stopping the thread.

[rcarriga]

This is part of nvim-dap, not neotest. You can use the following for debugpy, which will set the behaviour you want.

  dap.listeners.after.event_initialized["dap_exception_breakpoint"] = function()
    dap.set_exception_breakpoints({ "userUnhandled" })
  end

Background info can be found here https://github.com/microsoft/debugpy/issues/392. The behaviour seems a bit odd, but it's all handled by debugpy so any issues should be raised there

[wookayin]

I'm sorry for not having found microsoft debugpy issues (about userUnhandled): microsoft/debugpy#392, microsoft/debugpy#275, and microsoft/debugpy#111, indeed it seems a DAP issue. Exception breakpoints on non-test python files worked so I presumed this is a missing feature of neotest. Appreciate your help again.

[wookayin]

I found that the above set_exception_breakpoints({"userUnhandled"}) approach also often stops at "handled", "catched" exceptions (the code runs OK), but given that this is going to be a debugpy or DAP related issue I will try to follow up there. Just leaving a quick note for somebody who might find this in the future ...

UPDATE: Per :help dap.set_exception_breakpoints(), require'dap'.set_exception_breakpoints("default") would be good enough -- the argument is adapter-specific, for debugpy it would resolve to raised or uncaught exceptions.

UPDATE 2: Using the default filter would make the exception in the test case (caught by pytest) not hit a breakpoint. So neither userUnhandled nor raised is perfect -- let me keep track of the issue in microsoft/debugpy.

[wookayin]

I'm going to re-open it again and try to add some analyses because I think this is something that should be handled by neotest's pytest adapter, rather than debugpy/DAP itself. The reason is as follows:

userUnhandled is not suitable for normal use cases: it may hit a breakpoint even when unwanted

The userUnhandled exception breakpoint mode would break if the exception is not handled by user code: it will also stop at an exception that is catched and handled internally in a non-user-code. There can be a number of examples but one example would be:

import jax

DAP (debugpy) will hit a breakpoint at https://github.com/google/jax/blob/main/jax/_src/iree.py#L29 (import iree.compiler) due to an exception ModuleNotFoundError that is not handled by an user code:

which is however handled in the third-party library (see https://github.com/google/jax/blob/main/jax/_src/lib/xla_bridge.py#L41-L44):

try:
  import jax._src.iree as iree  # type: ignore
except (ModuleNotFoundError, ImportError):
  iree = None

Or when it crosses user code and library code:

class MyList(Sequence[int]):    

  def __len__(self):
     return 5
  def __getitem__(self, i: int):
    if i < 5:
      return i
    else:
     raise IndexError(i)  # <--------- raises IndexError if i >= len(self._list)  and debugpy breaks here when userUnhandled

def test_hello():
  s = [i for i in MyList()]
  assert s == [0, 1, 2, 3, 4]

because the exception crosses the boundary between user code and the built-in library, and is catched at https://github.com/python/cpython/blob/main/Lib/_collections_abc.py#L1015-L1023.

UPDATE: Later, I think this seems to be a debugpy issue where we may want to improve the behavior of userHandled. See https://github.com/microsoft/debugpy/issues/1102

The default filters (raised, uncaught) would NOT break at exceptions thrown from a test method

As explained in the original message of this thread, pytest will handle the exception thrown in the outermost code and simply print the test status: FAILED. What I want is to be able to break at the exception being thrown out of the pytest method.

pytest-pudb, for instance, works as a pytest adapter(plugin) and handles the exception and launch a PuDB session so that users can enter a post-mortem debugging. Since we are running a neotest_python adapter which runs pytest (https://github.com/nvim-neotest/neotest-python/blob/master/neotest_python/pytest.py#L118) through it, we could add a similar pytest plugin so that a raised exception can allow debugpy to kick in.

Challenge: interoperability?

A tricky part is that how to tell DAP (lua) to hit a post-mortem breakpoint at a specific traceback (python). I don't think there is such a DAP API yet; this would involve python-lua interoperability via RPC.

(neotest-python: python) -----> nvim-dap client (lua) -----> dap adapter (debugpy)
                           |                            |
                       Some API?                  Debug Adapter Protocol

Ideally, pydebug may have an API to enter a "post-mortem" debugging mode with the given traceback for the thrown exception (ref: microsoft/debugpy#722), similar to pytest-pudb (https://github.com/wronglink/pytest-pudb/blob/master/pytest_pudb.py#L100-L122).

Another alternative option is simply re-throw the exception outside pytest (when using the dap strategy) but this is so ugly and limited :|

If none of this works, the language-specific adapter debugpy could implement an exception breakpoint that would work only for those exceptions thrown out of pytest methods. I'm not sure yet this should be implemented on which side -- either debugpy or neotest-python.

[wookayin]

microsoft/debugpy#1103 discusses a way to implement post-mortem debugging mode. I wrote a PR #25 for this feature.

[zippeurfou]

@wookayin is the fix in master? I somehow still have the issue you pointed out. Not sure if I am missing anything in the conf.

[wookayin]

Yes I wrote #25 which works well and as expected (already on master). FYI here are some excerpt from my dap config:

DAP:

  require('dap-python').setup()
  require('dap-python').test_runner = 'pytest'

  -- Customize launch configuration (:help dap-python.DebugpyLaunchConfig)
  -- https://github.com/microsoft/debugpy/wiki/Debug-configuration-settings
  local configurations = require('dap').configurations.python
  for _, configuration in pairs(configurations) do
    -- makes third party libraries and packages debuggable
    configuration.justMyCode = false
    -- stop at first line of user code for better interaction.
    configuration.stopOnEntry = true
    -- dap-adapter-python does not support multiprocess yet (it often leads to deadlock)
    -- let's work around the bug by disabling multiprocess patch in debugpy.
    -- see microsoft/debugpy#1096, mfussenegger/nvim-dap-python#21
    configuration.subProcess = false

Neotest:

  require("neotest").setup {
    adapters = {
      require("neotest-python")({
        dap = {
          justMyCode = false,
          console = "integratedTerminal",
          stopOnEntry = false,  -- which is the default(false)
          subProcess = false,  -- see config/testing.lua
          openUIOnEntry = false,
        },
        args = { "-vv", "-s" },
        runner = 'pytest',
      }),
      require("neotest-plenary"),
    },
    ...
  }