Debug test support - Githubissues

lawrence-laz commented 7 months ago

Neotest supports DAP debugging for tests.

Checklist:

[x] Filter exact tests when starting the debug session
[x] Run build before debug session starts (preferably async)
[ ] In case of build errors, report errors to user and don't start debugging session
- Waiting for https://github.com/nvim-neotest/neotest/issues/415
[ ] Works with build.zig and without build.zig
[x] Support multiple outputs and in case of multiple prompt user to choose which to debug
[ ] By default all addTests artifacts from build.zig are named test. If there are multiple, we should warn user to add explicit name to their addTests.
[x] Allow overriding DAP configuration (ex. explicitly providing a debugger)
[ ] Autodetect available debugger if one is not provided in config (gdb, lldb, codelldb)

lawrence-laz commented 6 months ago

Partially implemented in de0bd238b7af47df2f2bea4db77512bb2bbf7caa Need to handle scenarios when build.zig is missing.

fnzr commented 5 months ago

I'm working on something related to this, particularly being able to debug individual tests. We can implement filtering tests without issues:

on your project build.zig, add the filter option to the test exe:

    const exe_unit_tests = b.addTest(.{
        .root_source_file = b.path("src/main.zig"),
        .target = target,
        .optimize = optimize,
        .filters = b.option([]const []const u8, "test-filter", "Skip tests that do not match any filter") orelse &[0][]const u8{},
    });

and then just add the parameter in the zig build command: https://github.com/fnzr/neotest-zig/blob/1a1800a95d534d9acdc5055dfeab8c8a84b7bdea/lua/neotest-zig/init.lua#L213

Both running and debugging individual tests is working on my machine, but the debug configuration is kinda iffy. I dont know what to put on the run_spec command, since both the build and run must be ran by the debugger (otherwise we'd be debugging an old exe), and I dont know how to capture compilation errors when running under dap.

But if the compilation works, we get step debugging on individual tests, whether they fail or not, which is really nice.

lawrence-laz commented 5 months ago

I was aware of the test-filter option, but for the following reasons chose to go with a custom test input file and filtering:

test-filter requires editing build.zig file, not a big deal, but creates additional friction, for example you cannot just zig init and debug the created tests immediatelly.
test-filter is not very good at filtering exact matches (last time I checked), it only does pattern matching, which triggers more tests than needed when their names overlap (ex. having two tests "My test" and "My test 2" and filtering for "My test" runs both of them). It also doesn't take into account that different files might have tests with the same name.

Instead I'd like debugging to use the same test runner that running tests uses now, where filtering is done via --neotest-input-path parameter and the contents of the input look something like this:

[
  {
    "source_path": "/Users/llaz/git/zig-enumerable/src/enumerable.zig",
    "test_name": "decltest.first",
    "output_path": "/var/folders/c2/yvzkyl496hn0bsc37tbc0fqc0000gn/T/nvim.llaz/QGJRcT/0"
  }
]

I had this mostly working in main as far as I remember, but didn't finish it yet.

Also, a few days ago I've become a father, so might be slow to respond, sorry about that!

fnzr commented 5 months ago

Ah, so that's what the input file does! I see, you're absolutely right that's the better approach. I think it's working as intended, I had no issues with it.

About debugging, I believe we need a better approach. Currently, the build step is made on the program function of the dap configuration:

program = function()
                vim.fn.system(
                    "zig build neotest-build --build-file neotest_build.zig -Dneotest-runner=\"/Users/llaz/git/neotest-zig/zig/neotest_runner.zig\"")
                return program_path
            end,

This almost works, but there are two issues:

1) program_path logic requires the user to have manually ran the build step before. This doesnt works for the first run, or after cleaning up cache/out dirs

2) if the build command fails, at best the dap execution will fail with "launch program not found" and at worst the dap will launch attached to an old version of the binary and you'll be debugging a program that is slightly off the code you're looking at.

I've tried searching, moving around the spec command, passing different dap configurations to the strategy, but nothing quite works reliably on error cases (but as I said, if the program builds fine everything just works).

There's the overseer plugin that allows executing a task before running tests, but adding a plugin dependency is obviously not ideal.

Congratulations on becoming a father, and don't worry about it, focus on your family!

lawrence-laz commented 5 months ago

You're right. I tried to summarize all that is left to do in regards to debugging at the top of this issue. Let me know if I missed anything.

It's not immediately obvious to me how to solve all of these either. We might have to look into how other neotest adapters for other languages have solved these issues.

fnzr commented 5 months ago

A bit of an update. I looked into reporting build errors and not launching debug session, but I dont think that's possible as is. Both golang and rust adapters suffer the same problem and launch the debugger even if the build fails. I opened an issue on neotest expanding on it. I think handling this situation requires either implementing a custom strategy or a upstream patch on neotest.

In the meantime, I completed a few tasks from this issue: async building, multiple output support (maybe), notification on build error (not ideal, but better) and user configurable debug adapter.

Can I make a pr with these or is it better to wait neotest response?

lawrence-laz commented 5 months ago

Please do! We don't have to resolve everything in a single go

lawrence-laz / neotest-zig

Debug test support #9