Is there a way to force stop running tests as soon as one test fails?

[alimoeeny]

In my project, I have a mixture of long-running integration tests and simple tests. If any of the simpler tests fail during the test run, I want to be able to halt all the tests immediately.

But a go test . command always goes through all tests and does not fail as soon as one Expect(1).To(Equal(2)) fails.

Am I ignoring something obvious? or is this the intended behavior?

Needless to say I am using gomega, without directly using ginkgo.

[onsi]

i think this is what you're looking for?

https://stackoverflow.com/questions/32046192/stop-on-first-test-failure-with-go-test

similar to ginkgo --fail-fast but for go test

[alimoeeny]

Thank you so much @onsi I didn't know about go test -failfast . very helpful.

In addition to that, in some scenarios, having something like panic that does not even wait for other tests that are mid flight to finish, would be very helpful.

Like I have these long running tests that talk to other processes and do rpc etc, and once started they would go on for a while. I don't know enough about internals of gomega to know if this is even feasible. But maybe a general flag or a new ExpectOrPanic or something.

In any case thank you very very much for the wonderful library you maintain.

And please feel free to close this issue.

[onsi]

thanks for the kind words!

The "stop the presses" behavior is something Ginkgo knows how to manage - but I'm not sure about go test. In Ginkgo, by default, a failure stops the current test immediately (and proceeds to clean up for that test) and, when running in parallel, --fail-fast will cause all other parallel specs to end immediately and clean up as well.

I'm not sure go test has that degree of orchestration when running in parallel. You can experiment with panicking or calling os.Exit yourself. You can tie that into Gomega with something like this:

func NewPanickingGomega() gomega.Gomega {
    return gomega.NewGomega(func(failure string, ...int) {
        panic(failure)
    }
}

now, in the tests you want to panic, instead of gomega.NewWithT(t) use NewPanickingGomega(). My hunch, though, is that go test will capture the panic and not abort the whole suite. So you might need something hokey like:

func NewExitingGomega() gomega.Gomega {
    return gomega.NewGomega(func(failure string, ...int) {
        fmt.Fprintln(os.Stderr, failure)
        os.Exit(1)
    }
}

but I'm not confident that you can os.Exit out of go test without some potential side effects (e.g. no cleanup code will be run).

FWIW - Ginkgo does solve for these pieces for you (and more!). I know it's DSL is not for everyone... and that there is a learning curve - but the level of complexity of the DSL you choose to adopt is up to you. In fact, I used to have a script that could take a go test suite and turn it into a Ginkgo suite by simply replacing func TestX(t *testing.T) { ... } with var _ = It("X", func() {t := GinkgoT() ...})

[alimoeeny]

Amazing, thank you very much. I need to experiment with the three suggestions and see what works best. I think

You are correct that I am unfamiliar with the Ginkgo DSL. Maybe I'll try it. For now, you gave me the idea I needed.

My tests are currently like:

func Test_saveAndGetTriggerInfo(t *testing.T) {
    RegisterTestingT(t)

        Expect(c).To(BeTrue())

I think I am going to replace all RegisterTestingT(t) with

RegisterFailHandler(func(failure string, args ...int) {
        fmt.Fprintln(os.Stderr, failure)
        os.Exit(1)
    })

some basic experimentation leads me to believe this is working for my use case. As you said there may be unexpected side effects that I will discover later.

Thank you @onsi

[alimoeeny]

@onsi so far this is a success. One minor problem is, on Fail, I don't get the line number for the failure, I still get the Expected X to equal Y message, but not the "file name + line number" Do you know why that is? where does that information come from in the default case?

[onsi]

hey @alimoeeny - ah makes sense. The line number is generated by the testing framework (either go test or Ginkgo) - by registering a custom fail handler you are sidestepping the framework and taking matters into your own hands. You can just print the stack out yourself (there are fancier approaches here that prune the stack and honor the optional skip parameter passed in, but this could be good enough):

RegisterFailHandler(func(failure string, args ...int) {
        debug.PrintStack() // just import go's debug package. this will print the current stack to stderr.  the line that failed will be in here somewhere.  you can look at debug.Stack() if you want to process the stack yourself.
    fmt.Fprintln(os.Stderr, failure)
    os.Exit(1)
})

[alimoeeny]

@onsi you are an open source hero. Thank you very much, I think my issue is properly and completely resolved.

For the record this is what I ended up doing

// from conversation with @onsi https://github.com/onsi/gomega/issues/660
// this makes it possible to fail fast and not wait for all the tests to finish if they are going to fail
func fastFail(failure string, args ...int) {
    // find the line number of the test that failed
    var culprit *runtime.Func
    var fileName string
    var line int
    for i := 1; i < 10; i++ {
        pc, _, _, ok := runtime.Caller(i)
        details := runtime.FuncForPC(pc)
        if ok && details != nil && !strings.Contains(details.Name(), "gomega") {
            culprit = details
            fileName, line = culprit.FileLine(pc)
            break
        }
    }

    if culprit != nil {
        fmt.Printf("called from %s\n%s:%d:", culprit.Name(), fileName, line)
    } else {
        fmt.Println("couldn't pinpoint the culprit")
    }

    //debug.PrintStack()
    fmt.Fprintf(os.Stderr, "\n%s %v", failure, args)
    os.Exit(1)
}