search

mehcode / config-rs

⚙️ Layered configuration system for Rust applications (with strong support for 12-factor applications).
Apache License 2.0
2.43k stars 206 forks source link

CI speed optimization #493

Closed matthiasbeyer closed 5 months ago

matthiasbeyer commented 8 months ago

This PR experiments with CI speedup optimizations.

I didn't realize that CI in this project did not even use caching... :eyes:

Closes #482 ... probably? :eyes:

matthiasbeyer commented 8 months ago

Some more changes, lets see how fast we're now. :laughing:

matthiasbeyer commented 8 months ago

Ok, I got 30 secs faster CI now... that's not nearly enough IMO. Could even be just random noise.

matthiasbeyer commented 8 months ago

I wonder why the tests for 1.70.0 is so much slower than the ones for 1.73.0 ... nextest did not actually make anything faster. Probably because building is slow, not running. But why is the job for 1.70.0 running 2:20min, and the 1.73.0 one only 58s? I don't know.

So overall we're now slower than before. :cry:

Let's see what happens if we execute tests right away and not wait for the check phase.

matthiasbeyer commented 8 months ago

4 min -> 3 min now.

Not sure whether this is actually worth it... but I guess I am ready for review here. Just a quick squash of the fixup commit.

polarathene commented 8 months ago

I wonder why the tests for 1.70.0 is so much slower than the ones for 1.73.0

1.73.0 being the latest stable right? I already mentioned that Github likely caches this themselves as they're known to do with other high traffic sources for CI. Far cheaper for them to pull from the real location once every N (eg: minute), than every CI job running within N duration on their infrastructure.

Older toolchains are less likely to have that optimization? 🤷‍♂️ (just an assumption, that there is something different in the requests that differentiates this)

I believe it's a common practice, even for ISPs to cache popular content from vendors like Youtube / Netflix (pretty sure I had read of partnerships to do such).

nextest did not actually make anything faster.

I also mentioned that observation, tests did not seem to take much of the time so I didn't see it providing much benefit.

So overall we're now slower than before

Keep in mind that when using actions/cache action, it will have access to main branch cache (once CI runs on that), and then PRs can use that. However PRs maintain their own scoped cache IIRC, and that is not accessible to another PR, nor main I think 🤷‍♂️

Should be in the docs, I could be recalling that wrong. Caching makes a big difference for docker-mailserver builds.

polarathene commented 8 months ago

As mentioned:

image

Additionally keep in mind what is being cached, and the cache key generation:

image

This also confirms that they're using the official github cache action, just extending it to better suite cargo:

image

We can see that the action is correctly saving and restoring cache quickly, but pay attention to this:

image

Your current PR is using the action, then setting a toolchain, which is against the actions advice.