search

Mbed-TLS / mbedtls-test

MbedTLS testing
Apache License 2.0
4 stars 23 forks source link

Unify Windows testing between PR and release jobs #27

Open gilles-peskine-arm opened 2 years ago

gilles-peskine-arm commented 2 years ago

The nightly/release jobs run more Windows builds than the PR job: the PR job only runs windows_testing.py with Visual Studio 2013 and mingw, whereas the nightly/release jobs test with all supported versions of Visual Studio (in 2.2x and 3.x, at the time of writing, that's 2015 and 2017). This is rarely a problem in practice because we rarely change Windows-specific things and we test with the oldest supported version, but it does introduce risk when merging a PR.

The goal of this task is to make Windows testing identical in all CI jobs, without meaningfully reducing the test coverage and without significantly increasing the typical lead time for a run of pr-head.

To keep the load moderate, we may want to do https://github.com/ARMmbed/mbedtls-test/issues/26 first.

Also, to keep the lead time for pr-head low, we may want to run some mostly-but-not-completely redundant tests only in pr-merge (and in non-PR jobs) and not in pr-head.

gilles-peskine-arm commented 1 year ago

https://github.com/Mbed-TLS/mbedtls/pull/7462 passed VS2013 (the pull request CI passed) but broke VS2015. This is the first time I remember seeing this happen.