Test runner integration

[chriskrycho]

Check for existing issues

• [X] Completed

Is your feature request related to a problem?

Other editors and IDEs (e.g. IntelliJ, Code, Visual Studio, Nova, Xcode, etc.) have the ability to put test results in a panel and often to run specific tests from a gutter indicator as well as the panel. Bonus: showing test results inline.

Describe the solution you'd like

• a pane for showing tests to be run, which can run one or more of them (preferably pluggable so it can work easily with different test runners for different languages)
• gutter integration for running individual tests (similarly)

If applicable, add mockups / screenshots to help present your vision of the feature

Here's how VS code presents a test you can run directly inline via the gutter:

And here's its testing panel:

[iamnbutler]

@chriskrycho From the design side this is a pretty giant task – A few questions:

• Who has the most intuitive test runner?
• Is there a minimal set of features to make a test runner useful, or is it an all or nothing sort of feature?
• What are the common pain points you have with existing test runners.

FYI we probably won't work on this immediately, just trying to get ahead of design here.

[chriskrycho]

@iamnbutler sorry for the delay in response—busy couple of weeks!

• Who has the most intuitive test runner? I don’t know that any of them are actually great, but/and they’ve all coalesced around similar designs. VS Code, IntelliJ, and Visual Studio all have a relatively similar design, and you can take the VS Code one as a reasonable representation of all of them. They usually include:

  • gutter integration to allow interacting with tests or test suites
  • a tree view showing the set of all the tests
  • some kind of reporting—minimally, via a terminal session, but more generally and more usefully, showing status on the gutter integrations and in the tree view

• Is there a minimal set of features to make a test runner useful, or is it an all or nothing sort of feature? I think the bare minimum is being able to run an individual test or a set of tests from the UI, whether from the gutter or from the tree view, and have that show the status of the test run in some way.

• What are the common pain points you have with existing test runners.

  • The VS Code integrations are very much test-runner specific. While there's a general set of plugin hooks, it's really up to the plugin implementer how they wire that up to the runner itself. My experience with Jest and Vitest here is… not great, to be honest, but those are sort of the best there is in the JS/TS world.

  • Reporting the results can be pretty janky. Doing it with popovers in the UI seems like a good idea, but (a) if that's too aggressive and actually moves your position in the editor, that's actively frustrating; and (b) if it's not very carefully designed it can end up just being impossible to actually get the relevant information out of it.

One additional note: I wouldn’t say it’s truly an MVP hard requirement to have the test runner integrate with debugger tools… but it’s pretty close. Debug-stepping through a failing test is super helpful.

[w01fgang]

For me having Wallaby.js is a game changer.

[ojkelly]

VS Code with Go has the best IDE testing feedback loop I've ever used. Everything else seems to have difference pieces of the puzzle, for me this includes the following:

• clickable run test link next to the test definition
• tree/list with all known tests in the project
• test output in a dedicated pane (zed's dock works pretty well for this when running tests via the cli)

And then the following are the game changers, that I don't think fit into a minimal feature set, but let me move faster:

• only run the tests which have changed, that is you edit the test, rerun that test but nothing else.
• capture coverage from the test and paint the lines covered in a subtle blue/red. This helps confirm you're testing the code you think you are—super helpful when you want to test all branches of a function.

I found line highlighting of test coverage much easier to grok that gutters. That might be a matter of preference between engineers, like debug-stepping which I've never found helpful.

I think most languages and frameworks can be supported without much extra work, if you're able to build on VS Code's Testing API (like the LSP).

[tomekowal]

Is there a minimal set of features to make a test runner useful, or is it an all or nothing sort of feature?

I believe Spacemacs has a pretty minimalistic approach

1. Define keybinding for
   • run test under cursor
   • run tests in the current file
   • run all tests in a project
   • re-run the previous test command (one of those above)
2. New buffer with test results (possibly could send a command to the terminal and reuse terminal window)
3. Allow clicking on stacktrace in test result to get back to code that produced it

It would probably save some time on adding the whole UI with test selection pane including checkmarks and buttons.

This minimal set of features enables quick test driven development:

1. Write a new test
2. Run and see it red
3. Iterate over code while constantly hitting "re-run previous test command"
4. See it green and go to 1.

[kellyfelkins]

Also, the ability to navigate between tests and test subjects.

In intellij, this is cmd-shift-t. It bounces you from the source file to the test file and vice versa. If the destination file is already open, moves you to that tab.

[baldwindavid]

The minimal keyboard-driven set of features that @tomekowal mentions are the only ones that I even care to use, so that's what I'd love to see as well. But really anything that makes it a little more possible to run a given test would be a welcome addition.

As of right now the closest I've found is editor::CopyRelativePath to copy the path and then paste alongside the test command in the terminal (e.g. mix test test/myapp/mytest.exs). Not ideal, but I can run a test file with this. A helpful small step might be adding a keybinding (e.g. editor::CopyRelativePathAndLineNumber) to get the path AND line number as test/myapp/mytest.exs:42 so that a test under cursor can be targeted.

[PJUllrich]

I mentioned this before in https://github.com/zed-industries/zed/issues/4913 but it would be great to be able to define my own keybindings to run 1. all tests in the current file 2. only the test currently focused on 3. all tests of the project.

In VSCode, I achieve this with custom tasks.json commands. If Zed would allow me to run CLI commands from custom keybindings, that'd fix already this issue for me.

[derekperkins]

One of my favorite features in Goland (IntelliJ) is that there isn't just top-level test integration, but per case integration if you are using table tests. I haven't seen that in VS Code, but maybe there's some plugin for it. That makes it even easier to run just an individual test by clicking the run icon in the gutter. https://www.jetbrains.com/help/go/performing-tests.html

[coolemur]

For VSCode I'm using https://marketplace.visualstudio.com/items?itemName=Orta.vscode-jest for Jest tests.

Well, it works. And has useful view of tests with links to each test location. Might be something for inspiration.

[tomekowal]

For those waiting for a proper test runner. I was able to work around it using commands in zed preview: In ~/.config/zed/tasks.json

  {
    "label": "test file",
    "command": "my-test-command",
    "args": ["$ZED_FILE"]
  },
  {
    "label": "test one",
    "command": "my-test-command",
    "args": ["$ZED_FILE", "$ZED_ROW"]
  }

and then in ~/.confing/zed/keymap.json

    "bindings": {
      ", t t": ["task::Spawn", { "task_name": "test one" }],
      ", t b": ["task::Spawn", { "task_name": "test file" }]
    }

That makes the issue less relevant for me.

[niclas-j]

For those waiting for a proper test runner. I was able to work around it using commands in zed preview: In ~/.config/zed/tasks.json

  {
    "label": "test file",
    "command": "my-test-command",
    "args": ["$ZED_FILE"]
  },
  {
    "label": "test one",
    "command": "my-test-command",
    "args": ["$ZED_FILE", "$ZED_ROW"]
  }

and then in ~/.confing/zed/keymap.json

    "bindings": {
      ", t t": ["task::Spawn", { "task_name": "test one" }],
      ", t b": ["task::Spawn", { "task_name": "test file" }]
    }

That makes the issue less relevant for me.

Thanks for the suggestion. Managed to get vitest to test the current file on a simple keybind.

  {
    "label": "Test current file",
    "command": "pnpm run test ${ZED_FILE}"
  }

Then you can just bind the "Test current file" task as you mentioned above.

[christian-gama]

I've found a way to easily run my tests. I can run tests from a file or directly from the test file. For example, if I run the script inside the users.controller.ts file, it will automatically find the test related to this file as long as you're following the idiomatic approach of naming a test file which in this case would be users.controller.(spec|test|e2e-spec).ts. If no test files are found with the expected format, it will then run all the tests in the current directory.

Then I created a shortcut to run the tests by simply pressing space r t (I thought something like r for run and t for test).

The script is located at .zed/ folder in my project workspace, so I can easily adjust the script to other languages or projects in the future.

.zed/test.sh

#!/bin/bash

filename=$ZED_FILE
dirname=$(dirname $filename)
basename=$(basename $filename)

cmd=pnpm

function makeArgs() {
    args=(run test --detectOpenHandles --silent=false --verbose $1)
    echo "${args[@]}"
}

base=${basename%.*}
base=${base%\.test}
base=${base%\.spec}
base=${base%\.e2e-spec}

for ext in e2e-spec.ts spec.ts test.ts; do
    if [ -f $dirname/$base.$ext ]; then
        echo $dirname/$base.$ext
        $cmd $(makeArgs $dirname/$base.$ext)
        exit $?
    fi
done

pnpm run test --detectOpenHandles --silent=false --verbose $dirname

exit $?

.zed/tasks.json

[
    {
        "label": "Test",
        "command": "$ZED_WORKTREE_ROOT/.zed/test.sh",
        "use_new_terminal": false,
        "reveal": "never"
    }
]

For the shortcuts, I'm using this (note that I use Vim, so you may need to change the context): .zed/keymap.json

{
    "context": "VimControl && !VimWaiting && !menu",
    "bindings": {
      "space r t": ["task::Spawn", { "task_name": "Test" }]
    } 
}

[kcamcam]

If this helps anyone this is how I got it working with RSpec (Ruby):

{
  "label": "RSpec",
  "command": "bundle exec rspec",
  "args": ["$ZED_FILE:$ZED_ROW"],
  "tags": ["rspec-test"]
}

[GrxE]

Here is my solution for just one test in Rust. Select the test-fn´s name and then it runs the selected test. Unfortunately it does not recognize if several tests are selected with a multiselection. Add in tasks.json :

{
    "label": "Run Test selected",
    "command": "cargo test $ZED_SELECTED_TEXT",
    "use_new_terminal": true,
    "allow_concurrent_runs": true
  }

and in keymap.json:

  {
    "context": "Workspace",
    "bindings": {
      "ctrl-r": ["task::Spawn", { "task_name": "Run Test selected" }]
    }
  }

[JosephTLyons]

Just to update this issue for any users who haven't been in Zed for awhile - we now have the ability to run tests in Zed, so long as your language extension provides the plumbing to inform Zed what a test is how to run it. You can run tests under your cursor, and we decorate the gutter with a run button next to the test. What we don't have yet is some test panel that shows all of the discovered tests.

[Scalahansolo]

On the latest version of Zed, I'm not seeing this for a Jest test in typescript. Are there specific issues(s) out there for various language services supporting this?

[paulyoung]

Can we customize keybindings for this?

[MusaGillani]

Just to update this issue for any users who haven't been in Zed for awhile - we now have the ability to run tests in Zed, so long as your language extension provides the plumbing to inform Zed what a test is how to run it. You can run tests under your cursor, and we decorate the gutter with a run button next to the test. What we don't have yet is some test panel that shows all of the discovered tests.

is there is anyway to change the default command in settings for the play button in the gutter?

[tindzk]

@MusaGillani Not to my knowledge, but you can change it in the sources. For example, I made the following local changes to use cargo-limit in Zed: https://gist.github.com/tindzk/4515c859182fa1bb9a0281582e303192

[Perfectio07]

Yeah the play button does not work for me because it spawns a new terminal and runs python3 without my poetry environment, thus immediately failing the test due to imports. I looked around and was not able to find any way to spawn new terminals inside the venv. Alternatively, if we could change the command somehow, I could just prepend it with poetry run python3 and that would make it happy but I have no clue if thats even possible

[nerder]

Just to update this issue for any users who haven't been in Zed for awhile - we now have the ability to run tests in Zed, so long as your language extension provides the plumbing to inform Zed what a test is how to run it. You can run tests under your cursor, and we decorate the gutter with a run button next to the test. What we don't have yet is some test panel that shows all of the discovered tests.

How can we configure that? I our mono-repo the entry point is different from the top level package.json and we need to be able to configure it to execute correctly

cc @JosephTLyons