cmd/go: go test -failfast doesn't stop immediately with t.Parallel()

timoha commented 5 years ago

What version of Go are you using (`go version`)?

$ go version
go version go1.10.7 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jenkins/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build781955494=/tmp/go-build -gno-record-gcc-switches"

What did you do?

func TestParallel1(t *testing.T) {
    t.Parallel()
    t.FailNow()
}

func TestParallel2(t *testing.T) {
    t.Parallel()
}

run with go test -v -failfast -parallel 1 returns:

=== RUN   TestParallel1
=== PAUSE TestParallel1
=== RUN   TestParallel2
=== PAUSE TestParallel2
=== CONT  TestParallel1
=== CONT  TestParallel2
--- PASS: TestParallel2 (0.00s)
--- FAIL: TestParallel1 (0.00s)
FAIL
exit status 1
FAIL    test    0.075s

What did you expect to see?

TestParallel2 not continue.

What did you see instead?

TestParallel2 continued execution.

bcmills commented 5 years ago

CC @jayconrod

bcmills commented 5 years ago

I doubt we'll get to this in Go 1.13, but I plan to do some work on the testing package for 1.14.

gopherbot commented 5 years ago

Change https://golang.org/cl/185877 mentions this issue: integration_test.go: use custom parallel test code

gopherbot commented 2 years ago

Change https://go.dev/cl/413236 mentions this issue: testing: skip parallel tests when -failfast is enabled

tommie commented 1 year ago

Another repro case, making this clearer:

package q

import "testing"

func TestA(t *testing.T) {
    t.Parallel()
    t.Fatal("a")
}

func TestB(t *testing.T) {
    t.Parallel()
    t.Fatal("b")
}

Yields:

$ go test -v -parallel=1 -failfast ./testing_test.go 
=== RUN   TestA
=== PAUSE TestA
=== RUN   TestB
=== PAUSE TestB
=== CONT  TestA
    testing_test.go:7: a
--- FAIL: TestA (0.00s)
=== CONT  TestB
    testing_test.go:12: b
--- FAIL: TestB (0.00s)
FAIL
FAIL    command-line-arguments  0.051s
FAIL

Though I would only expect one test to be marked "FAIL".

I think the issue is that Parallel() doesn't check shouldFailFast() after waitParallel() returns: https://github.com/golang/go/blob/2f6783c098696790223eae6986700868e9da0472/src/testing/testing.go#L1272

It's only checked early in Run(): https://github.com/golang/go/blob/2f6783c098696790223eae6986700868e9da0472/src/testing/testing.go#L1463

Considering this as the solution:

    t.context.waitParallel()

    if shouldFailFast() {
        if t.chatty != nil {
            t.chatty.Updatef(t.name, "=== ABORT %s\n", t.name)
        }
        t.SkipNow()
        return
    }

Edit Tests completed:

$ ../bin/go test -v -parallel=1 -failfast ./testing_test.go 
=== RUN   TestA
=== PAUSE TestA
=== RUN   TestB
=== PAUSE TestB
=== CONT  TestA
    testing_test.go:7: a
--- FAIL: TestA (0.00s)
=== ABORT TestB
--- SKIP: TestB (0.00s)
FAIL
FAIL    command-line-arguments  0.003s
FAIL

It would be nice to not have this "skipped", but it's better than actually running the test code. :)

golang / go