Cache Dependencies During GitHub Actions Build

[PhillyWebGuy]

@Romulus10 Unless someone wants to grab this, I wouldn't mind picking this up to keep me busy tomorrow nite.

[Romulus10]

Awesome, thanks! I just tagged you.

[isaacadams]

@PhillyWebGuy apologies if this is redundant with respect to research you've already collected, but I found this article to be really good -- towards the middle he begins suggesting tools and methods for optimizing caching and cli commands.

https://matklad.github.io/2021/09/04/fast-rust-builds.html

[isaacadams]

@Romulus10 is there any way we can isolate which pallet we are working on in any given CI such that it only runs checks for that pallet? If so, we could take advantage of that by making sure our pallets only point to the dependencies it uses + only worry about compiling/testing the pallet that has changed.

I suppose this could be done by leveraging the existing submodules and setting up CI for each pallet's repo.

[Romulus10]

I think that CI/CD for each pallet's repo is the only option there, but I'm not sure how much we'd need to do to get that working. Currently each pallet repo is JUST the pallet.

[PhillyWebGuy]

@Romulus10 Thanks for the link above. That's a good read and sounds like some good suggestions for ongoing improvements.

I'm currently reading up on this and seeing if this isapplicable to what we're doing with githab actions:

https://github.com/marketplace/actions/rust-cache

It looks like our action is doing this:

      - name: Check Build
        run: |
          SKIP_WASM_BUILD=1 cargo check --release

      - name: Test Build
        run: >
          cargo test

I didn't know what cargo check was, now I do:

Check a local package and all of its dependencies for errors. This will essentially compile the packages without performing the final step of code generation, which is faster than running cargo build. The compiler will save metadata files to disk so that future runs will reuse them if the source has not been modified. Some diagnostics and errors are only emitted during code generation, so they inherently won't be reported with cargo check.

It does compile. And the compiler saves metadata files, but I imagine they get lost on each build.

I'm thinking we might be able to use this. Doing more research at the moment though.

[Romulus10]

I'm actually wondering if it makes sense to skip the check phase. It's useful if you just need to verify, but I'm uncertain if it's necessary since we're immediately cargo testing

[PhillyWebGuy]

I'm trying to figure out this syntax, as it pertains to where to put that line of code. Is it this simple?

- name: Test Build
        uses: Swatinem/rust-cache@v1.3.0
        run: >
          cargo test

[PhillyWebGuy]

Might actually be, as a step:

- name: Enable caching
  uses: Swatinem/rust-cache@v1.3.0

[Romulus10]

I'm actually not completely sure - I think your second option is a fair shot, right before we start running Cargo commands.

[PhillyWebGuy]

Yeah, I lean towards the second. Do you have any idea how long the process took to run before? I guess the next step is to add that then run twice and compare it to your current benchmark, assuming it runs. We can do it whenever.

[Romulus10]

It runs in roughly half an hour.

[PhillyWebGuy]

Ok. Well, it might make sense to try it during the morning. It seems like you guys are committing at nite? Or we can do now. Your call. If it doesn't work, we just take it out, and no harm should be done.

[Romulus10]

As long as it runs in less than 29 minutes I count it as an absolute win.

[Romulus10]

Ok. Well, it might make sense to try it during the morning. It seems like you guys are committing at nite? Or we can do now. Your call. If it doesn't work, we just take it out, and no harm should be done.

The Actions will run on another branch, so we can try it out on a testing branch (let's call it milestone1/actions-optimize) whenever you're ready.

[PhillyWebGuy]

One sec...

[PhillyWebGuy]

[PhillyWebGuy]

That's what'd expect. We'll run it once, then again, and see what happens.

[Romulus10]

Awesome - thanks @PhillyWebGuy

[PhillyWebGuy]

[Romulus10]

Oh that is absolutely gorgeous.