spring-projects / spring-aot-smoke-tests

Smoke tests for Spring's AOT and native support
https://spring-team-aot-smoke-tests-dashboard.azuremicroservices.io/
51 stars 35 forks source link

Avoid GitHub sending build failure notifications for smoke tests that are not expected to pass #210

Closed wilkinsona closed 5 months ago

wilkinsona commented 6 months ago

We have a number of smoke tests that are not expected to pass. Right now, each daily run of these smoke tests fails and causes GitHub to send a notification:

Notifications for scheduled workflows are sent to the user who last modified the cron syntax in the workflow file

These notifications are unwanted noise as they aren't really telling us anything. It would be useful if we could configure, on a per-test basis, that the expected outcome is failure. Something like this in an individual test's build.gradle:

aotSmokeTest {
    expectedOutcomes {
        nativeAppTest = failure
        nativeTest = failure
    }
}

This would translate to setting continue-on-error in the relevant step of smoke-test-jvm.yml and/or smoke-test-native.yml using an input from the calling workflow.

If we wish, we could then go a step further and fail the build if a test passes unexpectedly by adding a subsequent step that fails if the step that ran the test passed unexpectedly.

wilkinsona commented 5 months ago

This would translate to setting continue-on-error in the relevant step of smoke-test-jvm.yml and/or smoke-test-native.yml using an input from the calling workflow.

Done in https://github.com/spring-projects/spring-aot-smoke-tests/commit/537c40976c7529796a73e304adace47cb4abbf07.

If we wish, we could then go a step further and fail the build if a test passes unexpectedly by adding a subsequent step that fails if the step that ran the test passed unexpectedly.

Done in https://github.com/spring-projects/spring-aot-smoke-tests/commit/e36fa988df3df19e4b78420f8e402400526bbf0e.