search

kubernetes-sigs / e2e-framework

A Go framework for end-to-end testing of components running in Kubernetes clusters.
Apache License 2.0
526 stars 101 forks source link

Test suite is executed even if the setup failed #423

Closed snorwin closed 4 months ago

snorwin commented 5 months ago

What happened?

The test suite is executed regardless of whether the setup actions were successful or not.

What did you expect to happen?

Tests should only be executed if all setup actions were successfully executed.

How can we reproduce it (as minimally and precisely as possible)?

It can be reproduced by adding a failing setup action to the test environment, e.g.:

testEnv.Setup(
    func(ctx context.Context, _ *envconf.Config) (context.Context, error) {
        return ctx, errors.New("this is an error")
    },
)

Anything elese we need to know?

No response

E2E Provider Used

kind

e2e-framework Version

v0.4.0

OS version

No response

vladimirvivien commented 5 months ago

@snorwin the framework tries to continue in case there are shutdown/finish logic to execute for a graceful shutdown. However, you can specify --fail-fast flag to force failures immediately https://github.com/kubernetes-sigs/e2e-framework/tree/main/examples/fail_fast.

snorwin commented 5 months ago

@vladimirvivien The --fail-fast option doesn't change this behavior. I agree graceful shutdown is needed, but running tests after a setup failure doesn't make sense.