[BUG] nested_tests causes test output from stdout/stderr to be printed undesirably

[fredrikaverpil]

Hi @rcarriga and thanks for this awesome project! 👋 I believe I have found a bug...

NeoVim Version

Output of nvim --version

NVIM v0.10.0-dev-2898+g4c31a1b80
Build type: RelWithDebInfo
LuaJIT 2.1.1710088188
Run "nvim -V1 -v" for more info

But this is also happening on 0.9.5 stable.

Describe the bug

When using nested_tests = true and when running nested tests, neotest will print out test result output from stdout/stderr, even when this is not desired.

For example, with neotest-go, which runs go test -json ... (produces JSON output), neotest-go will parse the JSON and present non-JSON output. However, neotest will also, undesirably, print out the original JSON output, which is not the intent. Unfortunately, this makes it very hard to traverse failing tests. See screenshot:

This does not seem to be a problem with neotest-go, as I'm writing my own neotest adapter for Go (fredrikaverpil/neotest-golang) and seeing the same behavior.

Workarounds

To work around this problem, you can make sure that the neotest.RunSpec command does not output anything to stdout/stderr and instead outputs a file to disk which path you attach to the neotest.Runspec context, so it can be read back and parsed in the neotest.Adapter.results method.

In my case, there is some minor output being made to stdout which causes the popup window to not show or show twice. I can get rid of this problem completely by immediately executing the following in my neotest.Adapter.results method, since I'm leveraging the workaround above (where I output all relevant test results to file):

vim.fn.writefile({ "" }, result.output)

However, I would prefer not to have to resort to such workarounds, as I cannot use the binary I want for the neotest.RunSpec command as it can not be "silenced" and will always output to stdout.

To Reproduce

If you wish, I can publish a small variant of neotest-go with far less complexity, which exhibits this problem. But in that case, and until then, you can just install neotest-go and clone down this repo of mine and run the same tests as you can see in the screenshot above.

1. git clone https://github.com/fredrikaverpil/go-playground.git
2. Install neotest-go.

   require("neotest").setup({
     adapters = {
       require("neotest-go")({
         experimental = {
           test_table = true,
         },
         args = { "-count=1", "-timeout=60s" }
       })
     }
   })

3. Make sure go is on your $PATH, e.g brew install go.
4. In neovim, open jsonoutput_test.go, place the cursor at the top level test and run it.
5. Open the test output panel and see how both unparsed JSON and parsed JSON is printed.