Explore using GitHub cache for build artifacts to save builder time, especially for things like incremental builds

[tt-rkim]

Inspired by cache usage, @TT-billteng , and #10878

Some ideas:

• We could use this: https://github.com/actions/cache?tab=readme-ov-file as a good starting point
• Need to be aware of security vulnerabilities such as: https://adnanthekhan.com/2024/05/06/the-monsters-in-your-build-cache-github-actions-cache-poisoning/
• Can introduce this in baby steps where we start off with an imprecise, but simple and correct, cache strategy where we try to reduce the scope of what can be cached with a simple key
• We can direct cache check and build artifact upload to GitHub runners rather than builders in order to further reduce time on builders. Although this may still have diminishing returns given that we only have 60 runners to begin with
• Other good practices: https://github.com/actions/cache/blob/main/caching-strategies.md

[tt-rkim]

@TT-billteng Looks like GitHub cache is only 10GB: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy

if we want to continue using GitHub cache. I'm seeing over >1GB on my local machine for all the object files, leaving us with a handful of possible entries to use before eviction starts.

I'm wondering if we should limit cache entries to contain binaries only from main branch in that case. This would hopefully speed up builds on branches anyway since we're working off a base of main. Even so, unless we think of ways to further reduce what we need to cache for incremental builds, only a couple of commits on main will actually have their artifacts cached at given time. I'm wondering how much this actually matters and if we should just go for it, still.

[tt-rkim]

Another limitation is we use the cache for python_env ! Oh no!

We're quite limited on space here. We may have to use our own storage for a caching solution.

[TT-billteng]

@TT-billteng Looks like GitHub cache is only 10GB: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy

if we want to continue using GitHub cache. I'm seeing over >1GB on my local machine for all the object files, leaving us with a handful of possible entries to use before eviction starts.

I'm wondering if we should limit cache entries to contain binaries only from main branch in that case. This would hopefully speed up builds on branches anyway since we're working off a base of main. Even so, unless we think of ways to further reduce what we need to cache for incremental builds, only a couple of commits on main will actually have their artifacts cached at given time. I'm wondering how much this actually matters and if we should just go for it, still.

If we have space just for one entry (latest main), that would be already a great start. Maybe one entry for each build variant (Release, Debug, RelWithDebInfo).

[tt-rkim]

Currently issue is mtime of precompiled headers: https://github.com/tenstorrent/tt-metal/actions/runs/10426638021/job/28880945967

Will try just removing them for now and seeing where we get.

[TT-billteng]

ran into this problem also when trying ccache ugh

[tt-rkim]

Trying to:

• keep trying on CI, but slow because limited builders
• setting up local https://github.com/nektos/act env to work faster in meantime

spinning off run with pcx and CMakeCache.txt deleted on CI right now

[tt-rkim]

First attempt at this with GitHub Cache didn't work too well and I closed my initial PR.

We can always revisit if needed.