golang / vscode-go

Go extension for Visual Studio Code
https://marketplace.visualstudio.com/items?itemName=golang.Go
Other
3.84k stars 735 forks source link

Test does not run correctly if subtest name contains regexp metacharacters #3070

Closed mattwelke closed 9 months ago

mattwelke commented 9 months ago

What version of Go, VS Code & VS Code Go extension are you using?

Version Information
* Run `go version` to get version of Go from _the VS Code integrated terminal_. - `go version go1.21.3 linux/amd64` * Run `gopls -v version` to get version of Gopls from _the VS Code integrated terminal_. - `v0.14.2` * Run `code -v` or `code-insiders -v` to get version of VS Code or VS Code Insiders. - `1.84.2` * Check your installed extensions to get the version of the VS Code Go extension - `v0.40.0` * Run Ctrl+Shift+P (Cmd+Shift+P on Mac OS) > `Go: Locate Configured Go Tools` command. - Output: ``` Checking configured tools.... GOBIN: undefined toolsGopath: gopath: /home/matt/go GOROOT: /usr/local/go PATH: /home/matt/.pyenv/shims:/home/matt/.pyenv/bin:/home/matt/.rbenv/shims:/home/matt/.rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/home/matt/.local/share/JetBrains/Toolbox/scripts:/usr/local/go/bin:/home/matt/cpu-temp-script:/home/matt/go/bin:/usr/local/bin/ go: /usr/local/go/bin/go: go version go1.21.3 linux/amd64 gopls: /home/matt/go/bin/gopls (version: v0.14.2 built with go: go1.21.3) gotests: /home/matt/go/bin/gotests (version: v1.6.0 built with go: go1.21.3) gomodifytags: /home/matt/go/bin/gomodifytags (version: v1.16.0 built with go: go1.21.3) impl: /home/matt/go/bin/impl (version: v1.1.0 built with go: go1.21.3) goplay: /home/matt/go/bin/goplay (version: v1.0.0 built with go: go1.21.3) dlv: /home/matt/go/bin/dlv (version: v1.21.2 built with go: go1.21.3) staticcheck: /home/matt/go/bin/staticcheck (version: v0.4.6 built with go: go1.21.3) go env Workspace Folder (golang-sandbox): /home/matt/golang-sandbox GO111MODULE='' GOARCH='amd64' GOBIN='' GOCACHE='/home/matt/.cache/go-build' GOENV='/home/matt/.config/go/env' GOEXE='' GOEXPERIMENT='' GOFLAGS='' GOHOSTARCH='amd64' GOHOSTOS='linux' GOINSECURE='' GOMODCACHE='/home/matt/go/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='linux' GOPATH='/home/matt/go' GOPRIVATE='' GOPROXY='https://proxy.golang.org,direct' GOROOT='/usr/local/go' GOSUMDB='sum.golang.org' GOTMPDIR='' GOTOOLCHAIN='auto' GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64' GOVCS='' GOVERSION='go1.21.3' GCCGO='gccgo' GOAMD64='v1' AR='ar' CC='gcc' CXX='g++' CGO_ENABLED='1' GOMOD='/home/matt/golang-sandbox/go.mod' GOWORK='' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' PKG_CONFIG='pkg-config' GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2574103532=/tmp/go-build -gno-record-gcc-switches' ```

Share the Go related settings you have added/edited

Run Preferences: Open Settings (JSON) command to open your settings.json file. Share all the settings with the go. or ["go"] or gopls prefixes.

"[go]": {
    "editor.wordWrapColumn": 100,
    "editor.rulers": [100, 120]
},

Describe the bug

I expected that when clicking on "Debug Test" for a subtest in the test explorer, with a valid breakpoint set, my breakpoint would be hit. Instead, my breakpoint isn't hit, and I get a message in the debug console telling me that there are "no tests to run".

Steps to reproduce the behavior:

Complete these steps as I did earlier...

I created a function and a test table for it using Go: Generate Unit Tests For Function from the command palette. I filled in the test table with one test that I expected to pass and one that I expected to fail. My code:

package main

import "fmt"

func sum(a, b int) int {
    return 2
}

func main() {
    a := 1
    b := 2
    fmt.Printf("The sum of %d and %d is %d.\n", a, b, sum(a, b))
}
package main

import "testing"

func Test_sum(t *testing.T) {
    type args struct {
        a int
        b int
    }
    tests := []struct {
        name string
        args args
        want int
    }{
        {
            name: "1_+_1_=_2",
            args: args{
                a: 1,
                b: 1,
            },
            want: 2,
        },
        {
            name: "1 + 2 = 3",
            args: args{
                a: 1,
                b: 2,
            },
            want: 3,
        },
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            if got := sum(tt.args.a, tt.args.b); got != tt.want {
                t.Errorf("sum() = %v, want %v", got, tt.want)
            }
        })
    }
}

I opened the test explorer tab:

image

I clicked "Run Test" on the root test (where my module name is). It then ran my tests:

image

I added a break point to my test body:

image

I clicked "Debug Test" just for that subtest/test table entry:

image

My breakpoint was not hit. The process finished and the debug console displayed:

Starting: /home/matt/go/bin/dlv dap --listen=127.0.0.1:35709 --log-dest=3 from /home/matt/golang-sandbox
DAP server listening at: 127.0.0.1:35709
Type 'dlv help' for list of commands.
testing: warning: no tests to run
PASS
Process 108182 has exited with status 0
Detaching

I clicked "Debug Test" at the test level instead of the subtest level:

image

This time, my breakpoint was hit, presumably because all of the subtests are being run now, in debug mode:

image

hyangah commented 9 months ago
$ go test
--- FAIL: Test_sum (0.00s)
    --- FAIL: Test_sum/1_+_2_=_3 (0.00s)
        main_test.go:35: sum() = 2, want 3
FAIL
exit status 1
FAIL    example/test    0.136s
$ go test -run=^Test_sum/1_+_2_=_3$
testing: warning: no tests to run
PASS
ok      example/test    0.131s

This is due to the use of + which is a regexp metacharacter. The extension has to scrub the discovered test name further to escape these.

mattwelke commented 9 months ago

Ah interesting. I ran into this issue with my actual test suite and when putting this minimal reproduction together for this report, I was able to reproduce it because I happened to also choose a character that was a regexp metacharacter. You can see I even tested whether the test name was related to the issue by changing my test name in this minimal reproduction from 1 + 1 = 2 to 1_+_1_=_2 (getting rid of spaces).

I don't recall what the test names were in my actual codebase, but if it would help with solving this issue I can go back to it and try this again, and report back.

adonovan commented 9 months ago

I fixed this (by chance this morning!) in https://go.dev/cl/546419 (see gopls/internal/server/command.go), when I noticed the code looked wrong. I confirmed the fix using VS Code and the gopls binary before/after this commit.

Run go install golang.org/x/tools/gopls@latest to try out the fix.

hyangah commented 9 months ago

Thanks Alan. But VS Code Go extension does not use gopls for its test and debug execution functionality. ~The relevant code location is, I think, https://github.com/golang/vscode-go/blob/322014f62edb664358415885cd0dc56a23249aaa/src/subTestUtils.ts#L21C19-L21C19~

p.s does gopls test codelens already support subtests? That will be great.

EDIT: the relevant code location is more deep in the test explorer run.ts. Keep looking...

adonovan commented 9 months ago

Thanks Alan. But VS Code Go extension does not use gopls for its test and debug execution functionality.

Funny, that's what I thought, but I tested the old and new gopls binaries, ABAB, and got consistent results. How strange.

p.s does gopls test codelens already support subtests? That will be great.

The codelens only lists top-level functions. But the the executeCommand(gopls.test) operation accepts any test name, including subtests, and passes it straight through to go test. I assumed that VS Code was using its native logic to figure out the names of all the subtests, but executeCommand to re-run a selected subtest.

findleyr commented 9 months ago

I assumed that VS Code was using its native logic to figure out the names of all the subtests, but executeCommand to re-run a selected subtest.

VS Code uses completely separate logic and implementation. At one point, there was thought of it using middleware to intercept the gopls code actions to identify subtests, and then pipe them into the native runner (so, the opposite of how you guessed it worked...).

adonovan commented 9 months ago

VS Code uses completely separate logic and implementation. At one point, there was thought of it using middleware to intercept the gopls code actions to identify subtests, and then pipe them into the native runner (so, the opposite of how you guessed it worked...).

Thanks for clarifying. I just checked again and now I can't reproduce what I saw before. Sorry for the confusion.

gopherbot commented 9 months ago

Change https://go.dev/cl/546260 mentions this issue: src/goTest: fix bugs in subtest handling