testify test suite test cases are not recognized, if import is missing in file

breml commented 3 years ago

Describe the bug

With the introduction of https://github.com/microsoft/vscode-go/pull/1707, support for running and debugging individual test methods for instances which use github.com/stretchr/testify/suite interfaces has been added to vscode-go.

To me it looks like this support has added only partially, because for me it only works, if the specific Go test file does include the import for "github.com/stretchr/testify/suite". If this is not the case, the commands "run test | debug test" are not shown.

Consider this setup (with two files in the same directory/package):

testify_test.go:

package mypkg_test

import (
    "github.com/stretchr/testify/suite"
)

type ExampleTestSuite struct {
    suite.Suite
}

func (suite *ExampleTestSuite) TestFoo1() {
    suite.True(true)
}

testify_test2.go

package mypkg_test

func (suite *ExampleTestSuite) TestFoo2() {
    suite.True(true)
}

In this case, the commands "run test | debug test" are show in the file testify_test.go for the method TestFoo1. But the same is not the case for the method TestFoo2 in the file testify_test2.go.

The problem can be resolved by adding the following import statement to testify_test2.go:

package mypkg_test

import (
    _ "github.com/stretchr/testify/suite"
)

func (suite *ExampleTestSuite) TestFoo2() {
    suite.True(true)
}

Screenshots

Without import:

With import:

What version of Go, VS Code & VS Code Go extension are you using?

Run go version to get version of Go
- go version go1.15.1 linux/amd64
Run code -v or code-insiders -v to get version of VS Code or VS Code Insiders
```
1.51.0
fcac248b077b55bae4ba5bab613fd6e9156c2f0c
x64
```
Check your installed extensions to get the version of the VS Code Go extension
- 0.18.1

hyangah commented 3 years ago

@breml Thanks for reporting the issue. The check for import statement was added because of the issue reported in microsoft/vscode-go/issues/1911. Unfortunately, the testify suite detection is done in the documentation level (https://github.com/golang/vscode-go/blob/8e1dadd138155988010c13c49682ac72fe64cb0b/src/testUtils.ts#L130-L166), not in the package level. And it uses only the pattern matching on the symbol name.

An ideal scenario is to ask gopls to provide symbol information accompanied with the type information, but I am not sure if we can achieve that with the standard LSP methods. (@stamblerre @suzmue)

jaybxyz commented 3 years ago

I'm also having the same issue. Can this issue be resolved? It seems like it's been quite a while.

firelizzard18 commented 2 years ago

Right now, the extension essentially looks at the file and asks, "Does this look like a test?" It has no knowledge of types or anything like that. This is done purely at the syntax layer. Due to these limitations, as long as test discovery is handled syntactically, I expect this issue to remain.

We have discussed moving test detection into gopls. This would be a far more robust way to detect tests, since gopls uses go/parser and friends, and thus has access to type information. Then, it would probably be straightforward to detect, "This type embeds suite.Suite, so it's methods are tests". But I don't think anyone is working on that.

golang / vscode-go