fix: Further improve CI runtime performance

[c12i]

The CI pipeline is experiencing significant slowdown when using Holonix, specifically:

• Initial environment setup takes considerable time
• Repeated operations across different jobs may be redundant
• Cache utilization might not be optimal

[ThetaSinner]

Suggestions for how to speed up the build:

• You should check whether crane is correctly caching the dependencies build. If you build the package, then change the code but not the Cargo.toml or the lock file, does it rebuild everything or just your code? If it rebuilds everything then take a look at the crane docs and make split the build into two steps.
• Once you start building Nix packages as part of the build, you need to set up your own cachix cache. Take a look at Wind Tunnel as an example. Or just the cachix build action docs.
• Only build the scaffolding once. You're depending on the CI shell which requires the scaffolding package. Build that once and push it to cachix then subsequent build jobs should be set to "needs" the step that's building the scaffolding package. That way they'll download it rather than rebuilding it. If you look at the build log timestamps, it's spending 1 minute downloading binaries and then ~15 minutes building the scaffolding.
• Add a Rust cache. For the build steps happening outside of Nix (just in the Nix shell, rather than a derivation), like the clippy and test, you can probably get a significant speed boost by adding something like the swatinem cache. We have that in several projects, so there are examples of how to use that.

Scaffolding depending on Holochain makes it quite slow to compile. It's a lot slower to build than Holochain, so I don't know if there is some dependency or configuration that we can tweak to speed up the compile times for scaffolding. It'd be worth looking into. But at least like this you can minimize the compiler work done to test,

[c12i]

Scaffolding depending on Holochain makes it quite slow to compile.

Yes this has been an issue I have tried to address by replacing the test_utils feature with hdk. Also replaced some holochain imports with holochain_types and removed the holochain_utils dependency entirely since it wasn't getting used as much. Still not enough it seems.

I think the dependency to holochain can be further stripped down. This the only part of the code that relies on an import from holochain, I think this is an essential import though since it's easier to rely on holochain to get the respective hdk and hdi versions without depending on them directly or hard coding the versions. Not sure whether these imports could be moved to holochain_types maybe?

Regarding the other suggestions you have provided. I will look into them and see how they can help

[c12i]

@ThetaSinner would you recommend I use the holochain-ci public cache for scaffolding? I noticed that you're using a private cache for wind tunnel

[ThetaSinner]

Use a private cache for CI builds. The public cache is now only populated for end-users to consume