search

golang / vscode-go

Go extension for Visual Studio Code
https://marketplace.visualstudio.com/items?itemName=golang.Go
Other
3.89k stars 753 forks source link

Testing tab shows confusing results when tests failing #1922

Closed mattwelke closed 2 years ago

mattwelke commented 2 years ago

What version of Go, VS Code & VS Code Go extension are you using?

Version Information
* Run `go version` to get version of Go from _the VS Code integrated terminal_. - go version go1.17.3 linux/amd64 * Run `gopls -v version` to get version of Gopls from _the VS Code integrated terminal_. - * Run `code -v` or `code-insiders -v` to get version of VS Code or VS Code Insiders. - v0.7.3 * Check your installed extensions to get the version of the VS Code Go extension - v0.29.0 * Run Ctrl+Shift+P (Cmd+Shift+P on Mac OS) > `Go: Locate Configured Go Tools` command. ``` Checking configured tools.... GOBIN: undefined toolsGopath: gopath: /home/matt/go GOROOT: /usr/local/go PATH: /home/matt/.vscode-server/bin/ccbaa2d27e38e5afa3e5c21c1c7bef4657064247/bin:/home/matt/.pyenv/shims:/home/matt/.rbenv/shims:/home/matt/.rbenv/bin:/usr/local/bin:/home/matt/.sdkman/candidates/maven/current/bin:/home/matt/.sdkman/candidates/java/current/bin:/home/matt/.sdkman/candidates/gradle/current/bin:/home/matt/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Python310/Scripts/:/mnt/c/Python310/:/mnt/c/Program Files/Common Files/Oracle/Java/javapath:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Windows/System32/OpenSSH/:/mnt/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR:/mnt/c/protoc-3.17.3-win64/bin:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files/Microsoft SQL Server/130/Tools/Binn/:/mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn/:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/ProgramData/chocolatey/bin:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Program Files/Git/cmd:/mnt/c/Users/mattw/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/mattw/AppData/Local/Programs/Microsoft VS Code/bin:/mnt/c/Users/mattw/.jdks/adopt-openjdk-11.0.11/bin:/mnt/c/Users/mattw/.dotnet/tools:/snap/bin:/home/matt/.dotnet/tools:/usr/lib/dart/bin:/home/matt/Downloads/dart-lang/protobuf/protoc_plugin/bin:/home/matt/n/bin:/usr/local/go/bin:/home/matt/go/bin:/home/matt/civo-cli:/home/matt/graal/graalvm-ce-java11-21.2.0/bin:/home/matt/kubectl-plugins go: /usr/local/go/bin/go: go version go1.17.3 linux/amd64 gopkgs: /home/matt/go/bin/gopkgs: go1.16.6 path github.com/uudashr/gopkgs/v2/cmd/gopkgs mod github.com/uudashr/gopkgs/v2 v2.1.2 h1:A0+QH6wqNRHORJnxmqfeuBEsK4nYQ7pgcOHhqpqcrpo= dep github.com/karrick/godirwalk v1.12.0 h1:nkS4xxsjiZMvVlazd0mFyiwD4BR9f3m6LXGhM2TUx3Y= dep github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= go-outline: /home/matt/go/bin/go-outline: go1.16.6 path github.com/ramya-rao-a/go-outline mod github.com/ramya-rao-a/go-outline v0.0.0-20210608161538-9736a4bde949 h1:iaD+iVf9xGfajsJp+zYrg9Lrk6gMJ6/hZHO4cYq5D5o= dep golang.org/x/tools v0.1.1 h1:wGiQel/hW0NnEkJUk8lbzkX2gFJU6PFxf1v5OlCfuOs= gotests: /home/matt/go/bin/gotests: go1.16.6 path github.com/cweill/gotests/gotests mod github.com/cweill/gotests v1.6.0 h1:KJx+/p4EweijYzqPb4Y/8umDCip1Cv6hEVyOx0mE9W8= dep golang.org/x/tools v0.0.0-20191109212701-97ad0ed33101 h1:LCmXVkvpQCDj724eX6irUTPCJP5GelFHxqGSWL2D1R0= gomodifytags: /home/matt/go/bin/gomodifytags: go1.16.6 path github.com/fatih/gomodifytags mod github.com/fatih/gomodifytags v1.13.0 h1:fmhwoecjZ5c34Q2chjRB9cL8Rgag+1TOSMy+grissMc= dep github.com/fatih/camelcase v1.0.0 h1:hxNvNX/xYBp0ovncs8WyWZrOrpBNub/JfaMvbURyft8= dep github.com/fatih/structtag v1.2.0 h1:/OdNE99OxoI/PqaW/SuSK9uxxT3f/tcSZgon/ssNSx4= dep golang.org/x/tools v0.0.0-20180824175216-6c1c5e93cdc1 h1:EAPsk8kfGCjxQagrkWjzXlUWe2p3gj5MknO+z2o9GKc= impl: /home/matt/go/bin/impl: go1.16.6 path github.com/josharian/impl mod github.com/josharian/impl v1.1.0 h1:gafhg1OFVMq46ifdkBa8wp4hlGogjktjjA5h/2j4+2k= dep golang.org/x/mod v0.2.0 h1:KU7oHjnv3XNWfa5COkzUifxZmxp1TyI7ImMXqFxLwvQ= dep golang.org/x/tools v0.0.0-20200522201501-cb1345f3a375 h1:SjQ2+AKWgZLc1xej6WSzL+Dfs5Uyd5xcZH1mGC411IA= dep golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4= goplay: /home/matt/go/bin/goplay: go1.16.6 path github.com/haya14busa/goplay/cmd/goplay mod github.com/haya14busa/goplay v1.0.0 h1:ED4BMrGQ3WH7H3YXrcnWMVzj1xeSepaYTkLh1DtFi/4= dep github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966 h1:JIAuq3EEf9cgbU6AtGPK4CTG3Zf6CKMNqf0MHTggAUA= dlv: /home/matt/go/bin/dlv: go1.16.6 path github.com/go-delve/delve/cmd/dlv mod github.com/go-delve/delve v1.7.0 h1:MaWAD3LtvjE/LL98urSHPjaMT+OubpQ2sqF3R2Uj1rc= dep github.com/cosiner/argv v0.1.0 h1:BVDiEL32lwHukgJKP87btEPenzrrHUjajs/8yzaqcXg= dep github.com/cpuguy83/go-md2man v1.0.10 h1:BSKMNlYxDvnunlTymqtgONjNnaRV1sTpcovwwjF22jk= dep github.com/google/go-dap v0.5.0 h1:RMHAVn5xeunBakYk65ggHXttk6qjZVdbmi+xhAoL2wY= dep github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc= dep github.com/mattn/go-isatty v0.0.3 h1:ns/ykhmWi7G9O+8a448SecJU3nSMBXJfqQkl0upE1jI= dep github.com/peterh/liner v0.0.0-20170317030525-88609521dc4b h1:8uaXtUkxiy+T/zdLWuxa/PG4so0TPZDZfafFNNSaptE= dep github.com/russross/blackfriday v1.5.2 h1:HyvC0ARfnZBqnXwABFeSZHpKvJHJJfPz81GNueLj0oo= dep github.com/sirupsen/logrus v1.6.0 h1:UBcNElsrwanuuMsnGSlYmtmgbb23qDR5dG+6X6Oo89I= dep github.com/spf13/cobra v0.0.0-20170417170307-b6cb39589372 h1:eRfW1vRS4th8IX2iQeyqQ8cOUNOySvAYJ0IUvTXGoYA= dep github.com/spf13/pflag v0.0.0-20170417173400-9e4c21054fa1 h1:7bozMfSdo41n2NOc0GsVTTVUiA+Ncaj6pXNpm4UHKys= dep go.starlark.net v0.0.0-20200821142938-949cc6f4b097 h1:YiRMXXgG+Pg26t1fjq+iAjaauKWMC9cmGFrtOEuwDDg= dep golang.org/x/arch v0.0.0-20190927153633-4e8777c89be4 h1:QlVATYS7JBoZMVaf+cNjb90WD/beKVHnIxFKT4QaHVI= dep golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae h1:Ih9Yo4hSPImZOpfGuA4bR/ORKTAbhZo2AbWNRCnevdo= dep gopkg.in/yaml.v2 v2.2.1 h1:mUhvW9EsL+naU5Q3cakzfE91YhliOondGd6ZrsDBHQE= dlv-dap: /home/matt/go/bin/dlv-dap: go1.16.6 path github.com/go-delve/delve/cmd/dlv mod github.com/go-delve/delve v1.7.1-0.20210804080032-f95340ae1bf9 h1:MNF+leMBsgJoT80cCoVJOsZWIJQlAOdTHgn0VslfJ6k= dep github.com/cosiner/argv v0.1.0 h1:BVDiEL32lwHukgJKP87btEPenzrrHUjajs/8yzaqcXg= dep github.com/cpuguy83/go-md2man/v2 v2.0.0 h1:EoUDS0afbrsXAZ9YQ9jdu/mZ2sXgT1/2yyNng4PGlyM= dep github.com/google/go-dap v0.5.0 h1:RMHAVn5xeunBakYk65ggHXttk6qjZVdbmi+xhAoL2wY= dep github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc= dep github.com/mattn/go-isatty v0.0.3 h1:ns/ykhmWi7G9O+8a448SecJU3nSMBXJfqQkl0upE1jI= dep github.com/peterh/liner v0.0.0-20170317030525-88609521dc4b h1:8uaXtUkxiy+T/zdLWuxa/PG4so0TPZDZfafFNNSaptE= dep github.com/russross/blackfriday/v2 v2.0.1 h1:lPqVAte+HuHNfhJ/0LC98ESWRz8afy9tM/0RK8m9o+Q= dep github.com/shurcooL/sanitized_anchor_name v1.0.0 h1:PdmoCO6wvbs+7yrJyMORt4/BmY5IYyJwS/kOiWx8mHo= dep github.com/sirupsen/logrus v1.6.0 h1:UBcNElsrwanuuMsnGSlYmtmgbb23qDR5dG+6X6Oo89I= dep github.com/spf13/cobra v1.1.3 h1:xghbfqPkxzxP3C/f3n5DdpAbdKLj4ZE4BWQI362l53M= dep github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= dep go.starlark.net v0.0.0-20200821142938-949cc6f4b097 h1:YiRMXXgG+Pg26t1fjq+iAjaauKWMC9cmGFrtOEuwDDg= dep golang.org/x/arch v0.0.0-20190927153633-4e8777c89be4 h1:QlVATYS7JBoZMVaf+cNjb90WD/beKVHnIxFKT4QaHVI= dep golang.org/x/sys v0.0.0-20210514084401-e8d321eab015 h1:hZR0X1kPW+nwyJ9xRxqZk1vx5RUObAPBdKVvXPDUH/E= dep gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= staticcheck: /home/matt/go/bin/staticcheck: go1.16.6 path honnef.co/go/tools/cmd/staticcheck mod honnef.co/go/tools v0.2.0 h1:ws8AfbgTX3oIczLPNPCu5166oBg9ST2vNs0rcht+mDE= dep github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ= dep golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4= dep golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4 h1:myAQVi0cGEoqQVR5POX+8RR2mrocKqNN1hmeMqhX27k= dep golang.org/x/tools v0.1.0 h1:po9/4sTYwZU9lPhi1tOrb4hCv3qrhiQ77LZfGa2OjwY= dep golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE= gopls: /home/matt/go/bin/gopls: go1.17.2 path golang.org/x/tools/gopls mod golang.org/x/tools/gopls v0.7.3 h1:Lru57ht8vtDMouRskFC085VAjBAZRAISd/lwvwOOV0Q= dep github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ= dep github.com/google/go-cmp v0.5.6 h1:BKbKCqvP6I+rmFHt06ZmyQtvB8xAkWdhFyr0ZUNZcxQ= dep github.com/sergi/go-diff v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0= dep golang.org/x/mod v0.4.2 h1:Gz96sIWK3OalVv/I/qNygP42zyoKp3xptRVCWRFEBvo= dep golang.org/x/sync v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ= dep golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e h1:WUoyKPm6nCo1BnNUvPGnFG3T5DUVem42yDJZZ4CNxMA= dep golang.org/x/text v0.3.6 h1:aRYxNxv6iGQlyVaZmk6ZgYEDa+Jg18DxebPSrd6bg1M= dep golang.org/x/tools v0.1.8-0.20211014194737-fc98fb2abd48 h1:hk7xRoeg0CG1nRLsd5BZLDUgVpA9bnKylGk1p2/BPH0= dep golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE= dep honnef.co/go/tools v0.2.0 h1:ws8AfbgTX3oIczLPNPCu5166oBg9ST2vNs0rcht+mDE= dep mvdan.cc/gofumpt v0.1.1 h1:bi/1aS/5W00E2ny5q65w9SnKpWEF/UIOqDYBILpo9rA= dep mvdan.cc/xurls/v2 v2.3.0 h1:59Olnbt67UKpxF1EwVBopJvkSUBmgtb468E4GVWIZ1I= go env Workspace Folder (pii-regex-test-go): /home/matt/code/github.com/mattwelke/pii-regex-test-go GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/home/matt/.cache/go-build" GOENV="/home/matt/.config/go/env" GOEXE="" GOEXPERIMENT="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOINSECURE="" GOMODCACHE="/home/matt/go/pkg/mod" GONOPROXY="github.com/groupby/*,github.com/mattwelke/*" GONOSUMDB="github.com/groupby/*,github.com/mattwelke/*" GOOS="linux" GOPATH="/home/matt/go" GOPRIVATE="github.com/groupby/*,github.com/mattwelke/*" GOPROXY="https://proxy.golang.org,direct" GOROOT="/usr/local/go" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64" GOVCS="" GOVERSION="go1.17.3" GCCGO="gccgo" AR="ar" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="/home/matt/code/github.com/mattwelke/pii-regex-test-go/go.mod" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2387387126=/tmp/go-build -gno-record-gcc-switches" ```

Share the Go related settings you have added/edited

Run Preferences: Open Settings (JSON) command to open your settings.json file. Share all the settings with the go. or ["go"] or gopls prefixes.

"go.toolsManagement.autoUpdate": true,
    "go.delveConfig": {
        "dlvLoadConfig": {
            "followPointers": true,
            "maxVariableRecurse": 1,
            "maxStringLen": 5000,
            "maxArrayValues": 100,
            "maxStructFields": -1
        },
        "apiVersion": 2,
        "showGlobalVariables": false,
        "debugAdapter": "legacy"
    },

Describe the bug

I set up some unit tests where I loop through an array of exactly 16000 strings and run a test for each string. I click the icon in the GUI to run the tests that is the left of the line numbers (because I want to see the test results displayed visually in the Testing tab on the left). Note that I do not click "run test" in the code editor because I don't want to see the concise results displayed, I want the verbose visual results where it shows the red and green icons.

The Go output displayed at the bottom shows me some cases where the tests failed and some where it passed. I know I have at least two failures. The Testing tab on the left though shows the number 15999 with a green icon. I'm not sure how to interpret this. But I think what it's telling me is that 15999 of my tests passed? But if so, this isn't correct. This is what it looks like:

image

Note how the output at the bottom proves I have at least two tests failing. I confirmed with the debugger that as it executes the tests, that slice has exactly 16000 strings:

image

Another thing I find strange is what is displayed to me in the Testing tab on the left as the tests execute. It has a few different numbers grow, and sometimes displays a red icon and sometimes a green icon:

go_test_vscode_problem2

Steps to reproduce the behavior:

  1. Create tests in Go.
  2. Run them by clicking on the icon to the left of func Test_... in the code editor so that the Testing tab shows the results visually.
  3. See the icon displayed (green checkmark) despite one of the tests failing, and a count that doesn't make sense.
mattwelke commented 2 years ago

Taking a closer look, I see now that there's another piece of information reported to me. It says 10638/16000 of my tests passed. This makes a lot more sense to me. I see about a 2/3 pass rate in the logs in the terminal too:

image

So at this point, my question is more about why one part of the UI says 10638/16000 passed and the other part of the UI says that 15999/16000 of them passed, and why it chose green checkmark icons for the tests where are least one failure occurred.

hyangah commented 2 years ago

my guess is that # mark go test uses to sanitize the subtest test name caused the confusion while encoding/decoding this test location into the uri (the assumed format is like https://github.com/golang/vscode-go/blob/50b13f17c9e93c29c12c37c2a9935757e0c08d38/src/goTest/resolve.ts#L171)

cc @firelizzard18

firelizzard18 commented 2 years ago

@mattwelke Every item in the tree view is a test, subtest, or directory (package, module, or workspace folder). The test controller is reporting "15999" as a subtest. It is not claiming that 15999/16000 of your tests succeeded. It should definitely be creating an entry for each subtest, not just the last one. I might need to implement a cap on the number of subtests, because adding 16000 items to the tree may cause issues. If you can point me to a git repo or provide me with a zip/tarball/etc that reproduces the issue, that would be very helpful.

In general, the test controller has problems around creating test items (nodes in the tree view). I've been using it a lot for the last few months, and it's definitely glitchy. Unfortunately, I'm working 50+ hours per week lately, so I don't really have bandwidth to look into this. Maybe over the holidays.

mattwelke commented 2 years ago

Hey @firelizzard18 I empathize with being busy. I'm in e-commerce so November and December are extra crazy. Don't worry about this for now. I'll put together a repro soon and we can work on it later on. This is a low priority for me.

Here, 16000 tests didn't mean I literally wrote 16000 tests and I'm interested in seeing each one in the tree. It's just a parameterized test with 16000 parameters. If I have one failure, I probably have a few hundred failures at least, so I didn't expect to get much value out of seeing the failed cases in that tree anyways.

I just wanted to report this in case it can be improved for the next person whose use case is more realistic than mine.

firelizzard18 commented 2 years ago

@mattwelke I was thinking more along the lines of, "Wow, what would happen if someone did something that created thousands of items in the tree? Probably nothing good, I should add a limit." The test controller is supposed to create an item in the tree for every time you call testing.T.Run, so if you call that 16000 times, it's supposed to be creating 16000 items. That would probably break VSCode, or at least make it unreasonably slow.

Originally the test controller always reported every line of test output as a separate output event. With one of my projects, if I enable logging, some of the tests create thousands of lines of output. That made VSCode very sad. Now (by default, it's configurable), if multiple lines of output are produced at the same line of code, the test controller concatenates them and reports it once.

Test item creation is definitely buggy, so the more different angles I can get on the problem, the better.

firelizzard18 commented 2 years ago

@hyangah would you tag this as WaitingForInfo?

mattwelke commented 2 years ago

@firelizzard18 Are you waiting on any particular information from me?

firelizzard18 commented 2 years ago

@mattwelke

I'll put together a repro soon and we can work on it later on.

hyangah commented 2 years ago

@gopherbot please add labels WaitingForInfo

(EDIT: don't know why but gopherbot seems to be on vacation)

@firelizzard18 I think this should work per https://github.com/golang/go/wiki/gopherbot and we are currently in process of sorting out the github permission to help project contributors. Thanks!

mattwelke commented 2 years ago

Work life balance is important. I'm not angry at GopherBot.

gopherbot commented 2 years ago

Change https://golang.org/cl/380501 mentions this issue: src/goTest: remove '#' in subtest name parsing cutset

gopherbot commented 2 years ago

Change https://go.dev/cl/380501 mentions this issue: src/goTest: remove '#' in subtest name parsing cutset

gopherbot commented 2 years ago

Change https://go.dev/cl/383935 mentions this issue: [release] src/goTest: remove '#' in subtest name parsing cutset

hyangah commented 2 years ago

cl/380501 (released in v0.31.1) should fix the bug that caused misinterpretation of subtests with name duplicates

Screen Shot 2022-02-14 at 5 58 01 PM

The only remaining task I see is to limit the number of tree items added to the ui. Created https://github.com/golang/vscode-go/issues/2068 to track that issue separately.