search

Mbed-TLS / mbedtls-test

MbedTLS testing
Apache License 2.0
4 stars 24 forks source link

Create a fail-fast CI stage #147

Open daverodgman opened 9 months ago

daverodgman commented 9 months ago

Create a serial fail-fast stage at the start of the CI pipeline which runs some quick sanity checks before launching the full CI. This might include:

The aim is to avoid using CI capacity for a PR with trivial issues.

gilles-peskine-arm commented 9 months ago

I disagree with failing fast if check_files or code_style fails. Those are trivial things and I don't want exotic configurations to be skipped just because one line is misintended.

I think the two cases when failing fast would do more good than harm is if the default config or the full config fails to build with clang (now that we do most builds with clang).

tom-cosgrove-arm commented 9 months ago

The thing is, if a fast check can fail quickly, to show that a second CI run is going to be needed, is it worth continuing that first check? If we think that there are slow checks that are likely to fail, so that a second CI run is also likely to fail (after the issues that caused the fast failure), then we likely have other problems

gilles-peskine-arm commented 9 months ago

The thing is, if a fast check can fail quickly, to show that a second CI run is going to be needed, is it worth continuing that first check?

Yes! Each CI runs adds significant latency in the development process, so the more feedback we have, the better. The reason I push an update to the CI is to get testing in the hundred or so combinations of configuration options and compiler versions. We already do get quick feedback for quick checks: the results can be seen in real time on the CI server. If the failure of a quick check interrupts the build, another cycle is needed.

Most quick checks are easy to resolve, and have a resolution that's mostly independent from other concerns. Failures in exotic configurations can mean design issues (oops, I hadn't thought of this edge case) or difficult investigations with a debugger (how is this case reachable???). A report of exotic configurations is useful early in the development process. Fixing the quick checks can usually be a late patch.

Failing fast reduces the CI load a little, at the expense of developer time. Developer time is more expensive. It only makes sense to fail fast if the CI results would be mostly useless anyway.