search

kubernetes-sigs / e2e-framework

A Go framework for end-to-end testing of components running in Kubernetes clusters.
Apache License 2.0
529 stars 103 forks source link

After panic recovery and performing of finish steps tests are not marked as failed #328

Closed StupidScience closed 1 year ago

StupidScience commented 1 year ago

What happened?

After panic recovery and performing of finish steps tests are not marked as failed

What did you expect to happen?

After panic I expect tests to be marked as failed ones

How can we reproduce it (as minimally and precisely as possible)?

func TestMain(m *testing.M) {
    testEnv := env.New()
    testEnv.Setup(
        causePanic(),
    )

    os.Exit(testEnv.Run(m))
}

func causePanic() env.Func {
    return func(ctx context.Context, config *envconf.Config) (context.Context, error) {
        panic("aaaaa")

        return ctx, nil
    }
}

this code will cause panic and recover but exit code of go test command will be 0 and tests will be reported as ok

Anything elese we need to know?

No response

E2E Provider Used

kind

e2e-framework Version

0.3.0

OS version

any

harshanarayana commented 1 year ago

@StupidScience This is a nice catch. Thanks for bug report. 

❯ go test -v .
E1026 17:27:03.045149   58234 env.go:361] Recovering from panic and running finish actions: aaaaa, stack: goroutine 1 [running]:
runtime/debug.Stack()
    /Users/harshnar/.gvm/gos/go1.19/src/runtime/debug/stack.go:24 +0x65
sigs.k8s.io/e2e-framework/pkg/env.(*testEnv).Run.func1()
    /Users/harshnar/Work/Tesseract/Maglev/mks/k8s/e2e-framework/pkg/env/env.go:361 +0x9b
panic({0x2075d40, 0x24d1f30})
    /Users/harshnar/.gvm/gos/go1.19/src/runtime/panic.go:884 +0x212
sigs.k8s.io/e2e-framework/examples/panictest.TestMain.func1({0xc000171c98?, 0x1050e74?}, 0x50?)
    /Users/harshnar/Work/Tesseract/Maglev/mks/k8s/e2e-framework/examples/panictest/main_test.go:23 +0x27
sigs.k8s.io/e2e-framework/pkg/env.(*action).run(0x21972e0?, {0x24ea050?, 0xc000128008?}, 0xc00048c800)
    /Users/harshnar/Work/Tesseract/Maglev/mks/k8s/e2e-framework/pkg/env/action.go:135 +0xea
sigs.k8s.io/e2e-framework/pkg/env.(*testEnv).Run(0xc000171e88, 0xc0004a86b0?)
    /Users/harshnar/Work/Tesseract/Maglev/mks/k8s/e2e-framework/pkg/env/env.go:379 +0x217
sigs.k8s.io/e2e-framework/examples/panictest.TestMain(0xffffffffffffffff?)
    /Users/harshnar/Work/Tesseract/Maglev/mks/k8s/e2e-framework/examples/panictest/main_test.go:18 +0x9d
main.main()
    _testmain.go:49 +0x1d3
FAIL    sigs.k8s.io/e2e-framework/examples/panictest    1.420s
FAIL

I believe this is what you are expecting to happen ?

StupidScience commented 1 year ago

@harshanarayana hard to tell but on first sight looks correct.