Misidentifying ginkgo CLI version. States V1/V2 compatibility issue.

[orsenthil]

I am using IntelliJ IDEA 2023.1.2 Preview (Ultimate Edition)

I am trying to run this test https://github.com/aws/amazon-vpc-cni-k8s/blob/master/test/integration/cni/service_connectivity_test.go

I have ginkgo version 2.

$ ginkgo version
Ginkgo Version 2.1.3

And the tests run fine, if I exercise them using CLI (In the fact the same command below).

/usr/local/go/bin/go run github.com/onsi/ginkgo/v2/ginkgo -v -v --fail-on-pending -- --cluster-kubeconfig=$KUBECONFIG --cluster-name=$CLUSTER_NAME --aws-region=$AWS_REGION --aws-vpc-id=$VPC_ID --ng-name-label-key=$NG_NAME_LABEL_KEY --ng-name-label-val=$NG_NAME_LABEL_VAL "--focus=[STATIC_CANARY] test service connectivity"

But the plugin is throwing an error with V1/V2 compatibility issue. How should I resolve this?

An error occurred with ginkgo CLI this usually is a V1/V2 compatibility issue. 
Please make sure the ginkgo CLI version matches the version used by your project. 
You can install the appropriate CLI by running 'go install github.com/onsi/ginkgo/ginkgo@v1' or 'go install github.com/onsi/ginkgo/v2/ginkgo@v2' 

[TaylorOno]

That error is pretty generic meaning the ginkgo CLI behave unexpectedly,

The ginkgo plugin should generate an run this command

/usr/local/go/bin/go run github.com/onsi/ginkgo/v2/ginkgo  -v "--focus=[CANARY] test service connectivity"

it should be visible in the run dialog after the setup calls. ie.

Can you confirm the command that is being generated and that the generated command is able to be execute on the command line?

I'll have to dedicate some time to testing against the 2023.1.2-preview version. In the mean time you can executing with debug which will side step the ginkgo CLI.

[nvictor]

Hello,

I would like to bump this issue up because I am seeing the same exact thing with Goland and ginkgo 2.13.1.

If the error message is pretty generic, how can we tell what exactly is going on?

[nvictor]

Additional information. These two lines in the code are what seem to be causing the problem:

https://github.com/IdeaGinkgo/Intellij-Ginkgo/blob/master/src/main/java/com/github/idea/ginkgo/GinkgoTestEventsConverterV2.java#L21

and

https://github.com/IdeaGinkgo/Intellij-Ginkgo/blob/master/src/main/java/com/github/idea/ginkgo/GinkgoTestEventsConverter.java#L91

I don't know why these checks are written that way but when I run ginkgo -r -v (version 2.13.1) on the command line with I do get lines starting with === RUN.

[TaylorOno]

@nvictor these checks were written because of a breaking change to capture the v1/v2 breaking changes. It was a naive approach at the time if you tried to run v1 tests with the v2 CLI or vice versa. This is somewhat solved by the introduction of go tools in ginkgo v2.

What OS are you using? I don't see that behavior on windows the first line is the suite that is being ran which is actually used here https://github.com/IdeaGinkgo/Intellij-Ginkgo/blob/07a607d14079dab2c60ea1d9a8fa2a85d30e2401/src/main/java/com/github/idea/ginkgo/GinkgoTestEventsConverter.java#L112

PS C:\Users\taylor\workspace\amazon-vpc-cni-k8s\test\integration\cni> C:/Users/taylor/go/bin/ginkgo.exe version
Ginkgo Version 2.13.2
PS C:\Users\taylor\workspace\amazon-vpc-cni-k8s\test\integration\cni> C:/Users/taylor/go/bin/ginkgo.exe -r -v                                          
Running Suite: CNI Pod Networking Suite - C:\Users\taylor\workspace\amazon-vpc-cni-k8s\test\integration\cni
==========================================================================================================
Random Seed: 1703325584

Will run 17 of 17 specs
------------------------------

If the suite doesn't appear I'll need to make modifications to the output parser would you be able to share the full output you are seeing

[nvictor]

Hey Taylor, I really appreciate you looking into this. I wouldn't want you to work during the holidays as this is not urgent at all 😊

Also, we may need a better check but let's confer with the ginkgo folks first. They might know what a better check in this case could be.

Now, the issue is with mixing regular go tests and ginkgo tests. I am able to reproduce the problem with the following setup. I was on macos, but I don't think the issue is with the OS.

Thank you

➜  scratch tree .                 
.
├── blah
│   ├── blah.go
│   └── blah_test.go
├── go.mod
└── go.sum

2 directories, 4 files
➜  scratch cat blah/blah.go
package blah

func GetBlah() string {
    return "blah"
}
➜  scratch cat blah/blah_test.go 
package blah_test

import (
    "testing"

    "example.com/scratch/blah"
)

func TestBlah(t *testing.T) {
    t.Run("First test", firstTest)
}

func firstTest(t *testing.T) {
    got := blah.GetBlah()
    expected := "blah"
    if got != expected {
        t.Errorf("blah.GetBlah() = %s; want %s", got, expected)
    }
}
➜  scratch /Users/victor/go/bin/ginkgo version
Ginkgo Version 2.13.1
➜  scratch /Users/victor/go/bin/ginkgo -r -v
=== RUN   TestBlah
=== RUN   TestBlah/First_test
--- PASS: TestBlah (0.00s)
    --- PASS: TestBlah/First_test (0.00s)
PASS

Ginkgo ran 1 suite in 709.562851ms
Test Suite Passed

[TaylorOno]

If I import ginkgo to you example test test output changes. But in the mean time I can remove the check for ===RUN in the v2EventParser since it shouldn't be needed.

PS C:\Users\taylor\GolandProjects\awesomeProject> cat .\blah_test.go
package blah

import (
        _ "github.com/onsi/ginkgo/v2"
        "testing"
)

func TestBlah(t *testing.T) {
        t.Run("First test", firstTest)
}

func firstTest(t *testing.T) {
        got := GetBlah()
        expected := "blah"
        if got != expected {
                t.Errorf("blah.GetBlah() = %s; want %s", got, expected)
        }
}
PS C:\Users\taylor\GolandProjects\awesomeProject> ginkgo -r -v
PASS

Ginkgo ran 1 suite in 2.287081s
Test Suite Passed

[nvictor]

Thanks Taylor 😊

[TaylorOno]

Were you still having this problem or can I close the issue for now?

[nvictor]

Happy New Year Taylor :tada:

I am sorry for the late reply. I just got back to work this week. The issue still persists on my work machine even with the new extension version 0.10.3.

Not a big deal since we identified what the real problem is. What's concerning though is that your changes doesn't seem to have any effect.

Is there a way for me to verify the code for the extension itself that my IDE is using?

[TaylorOno]

if you see v0.10.3 in the plugin store that should be the code it is using. I'll have to create a better way to identify the version internally as well as add a flag to get more debug data out.

[cseatjc]

I am running into the issue as well but only on debug. Ginkgo version 2.13.2, with latest plugin version.

We use custom build flags like this:

//go:build integration
// +build integration

package drivers_test

import (
    "context"
    "time"

    . "github.com/onsi/ginkgo/v2"
    ...

And in the ginkgo additional options section of the config I put: --tags integration --label-filter integration

This results in the same error message described above but only when debugging. On run everyting works as expected. Taking the build tags out of the code and removing those additional options results in debugging working as expected.

Even if these tags aren't supported by the plugin for some reaseon, the error message should at least be updated to be more helpful as I spent hours trying to figure out what was wrong with my version of ginkgo.

[TaylorOno]

@cseatjc you won't be able to debug with the --label-filter you would have to use --ginkgo.label-filter integration in the ginkgo additional arguments sections but I will be removing that guard class and logging the CLI response

[cseatjc]

Hmm @TaylorOno that gives me

/opt/homebrew/Cellar/go/1.21.6/libexec/bin/go run github.com/onsi/ginkgo/v2/ginkgo -v --tags integration --ginkgo.label-filter integration --focus=integration

ginkgo run failed
  flag provided but not defined: -ginkgo.label-filter

Here is my config: