search

exercism / rust-test-runner

GNU Affero General Public License v3.0
3 stars 15 forks source link

Use sccache to improve build time #35

Open dhovart opened 2 years ago

dhovart commented 2 years ago

sccache can be used to cache built dependencies and improve solutions build time.

I may work on this this week.

dhovart commented 2 years ago

@ErikSchierboom I don't know much of your infrastructure. Is it possible to use if it exists a docker volume for storage that would store:

1/ the compilation cache from sccache 2/ and additionally the rust local registry introduced in #27, that would contain older versions of a crate (instead of regenerating it each time with the crates newest versions as we do each time we build the test runner) ?

ErikSchierboom commented 2 years ago

@dhovart Currently we do not support external volumes. We might in the future.

ErikSchierboom commented 2 years ago

@dhovart Couldn't you build an exercise at Docker build time (as in: in the Dockerfile) that uses all dependencies? I imagine that the compilation cache is then populated and that, when running the image, it will re-use that.

senekor commented 1 year ago

This may as a side effect allow people to use compile-time heavy crates, e.g. anything syn-based. Currently, using such dependencies makes the tests time out due to long build time.

ErikSchierboom commented 1 year ago

This may as a side effect allow people to use compile-time heavy crates, e.g. anything syn-based. Currently, using such dependencies makes the tests time out due to long build time.

Can we precompile those?

senekor commented 1 year ago

I thought that was the point of using sccache, caching the compilation of dependencies. I haven't used it before myself though, so I might be misunderstanding something.

ErikSchierboom commented 1 year ago

I thought that was the point of using sccache, caching the compilation of dependencies. I haven't used it before myself though, so I might be misunderstanding something.

Oh I don't know much about Rust. What does sccache do exactly?

senekor commented 1 year ago

It caches build artifacts in a more sophisticated manner than the standard toolchain does. C and C++ are supported too.

Currently, every exercise that's submitted is compiled from scratch, including its dependencies. This is both time consuming (costly) and actually prevents some dependencies from being used at all, because they increase build time so much the timeout is triggered.

The proposed solution is to use sccache to precompile the dependencies, embed that cache in the docker image and compile submitted solutions through sccache as well.

I think this would be very beneficial and hope to find time to pick up the stale PR at some point.

(I wonder how much space that cache might take, since we're also talking about reducing the image size atm.)

dhovart commented 1 year ago

Hi, sorry for the PR going stale. Let me know if you'd want me to take this over again.

senekor commented 1 year ago

@dhovart that sounds great! Reading the description of that PR, it kinda depends on this one: https://github.com/exercism/rust-test-runner/pull/40

I believe the last state of that PR was to try to use dependabot to automatically update version-pinned dependencies. Would you be interested in tackling that?

ErikSchierboom commented 1 year ago

The proposed solution is to use sccache to precompile the dependencies, embed that cache in the docker image and compile submitted solutions through sccache as well.

That sounds awesome.

dhovart commented 1 year ago

@dhovart that sounds great! Reading the description of that PR, it kinda depends on this one: #40

I believe the last state of that PR was to try to use dependabot to automatically update version-pinned dependencies. Would you be interested in tackling that?

Thanks for your work on this. I actually won't have time to continue working on this one until next week, so feel free to do it if you want to start today!