Make device tests faster in CI

grdddj commented 1 year ago

Possible improvements:

split the CI job into more parts
run btc-only job with regular firmware and only altcoin stuff in "regular" job
not having so many jobs in build pipeline, so the tests start earlier (as frozen debug has bigger chance of finishing earlier)
- could do build_for_tests pipeline that would need to finish before all other (not-so-important) build would start - could put rest of build after the test phase
optimize tests themselves - shorten preparation time, increase same client reuse, investigate how to quicken UI tests

Results of time-measurements: Device tests: https://gist.github.com/grdddj/cc13884dd75350ad47deddef8f06ccf1 UI tests: https://gist.github.com/grdddj/56a590bef63de455d16bf440e23e1e4e

matejcik commented 1 year ago

not having so many jobs in build pipeline, so the tests start earlier (as frozen debug has bigger chance of finishing earlier)

i believe each job only waits for its dependencies? i.e. you can have "test" running while "build" is still going on

grdddj commented 1 year ago

not having so many jobs in build pipeline, so the tests start earlier (as frozen debug has bigger chance of finishing earlier)

i believe each job only waits for its dependencies? i.e. you can have "test" running while "build" is still going on

Yes, some test jobs start earlier, but I noticed the frozen debug build which we need is much quicker when there are no other builds going on - https://gitlab.com/satoshilabs/trezor/trezor-firmware/-/jobs/3849597107 vs https://gitlab.com/satoshilabs/trezor/trezor-firmware/-/jobs/3850730600 (3 vs 6 minutes). So it would be nice to build this one first, so device tests can start ASAP

matejcik commented 1 year ago

partially done via #2857, more work on this is low priority at this time

trezor / trezor-firmware

Make device tests faster in CI #2855