Add Spack buildcache to CPU Ubuntu builds

[cameronrutherford]

Merge request type

• [x] New feature

Relates to

• [x] Spack configuration
• [x] Other

This MR updates

• [x] Spack configuration
• [x] Other

Summary

Adds buildcache support for ExaGO Spack CPU builds. Should substantially speed up builds, and enable adding more of these builds in the future.

[cameronrutherford]

Good to merge with an approving review.

Note that this has moved our spack CPU build speeds down from 1.5hr to <20 mins

[abhyshr]

Might be better for Jaelyn or Ryan to review this. I am not familiar with this stuff.

[cameronrutherford]

https://asciinema.org/a/KCi5TmUXc6zWDj7JYHzfSFxmw - I need to update docs, but now installing exago based on these buildcaches takes <60 seconds :)

[cameronrutherford]

I took a sweep at the documentation and it 100% needs a review. We should probably replace some of my lazier language with just issue references and get more help cleaning up some of the things that are more out of date.

@jaelynlitz since we are really pushing jupyter tutorials in the future, quarto will be really nice for sustainability, and making things prettier + easier to maintain. I think our pdf is probably extremely out of date in a few respects now.

[cameronrutherford]

@abhyshr take a look at my documentation additions to the branch at least

[ryandanehy]

Not sure this is the place for this comment, but there is a link in the buildsystem readme.md that points to gitlab which of course requires a login. Since this is a public repo should we refrain from linking to gitlab in our docs? It links to an old MR.

[ryandanehy]

Not sure this is the place for this comment, but there is a link in the buildsystem readme.md that points to gitlab which of course requires a login. Since this is a public repo should we refrain from linking to gitlab in our docs? It links to an old MR.

broken link here as well https://github.com/pnnl/ExaGO/blob/spack-ubuntu-cache-dev/buildsystem/spack/README.md

[ryandanehy]

Broken links on this page https://github.com/pnnl/ExaGO/blob/spack-ubuntu-cache-dev/docs/exago_policy_compatibility.md We might need to open PR later to deal with the revamp / refresh of the docs

[cameronrutherford]

@ryandanehy I fixed most of your link suggestions, and I have left the one GitLab link in there

[jaelynlitz]

my other minor comment would be that I want the spack spec earlier in the pipeline name with less text in front of it so I can see the whole spec in the check status list

[cameronrutherford]

my other minor comment would be that I want the spack spec earlier in the pipeline name with less text in front of it so I can see the whole spec in the check status list

See if newer pipelines are any better. We can also shorten initial name to fit more of spack spec.

[jaelynlitz]

my other minor comment would be that I want the spack spec earlier in the pipeline name with less text in front of it so I can see the whole spec in the check status list

See if newer pipelines are any better. We can also shorten initial name to fit more of spack spec.

That looks better, I can now see the end of the spec to see the variance in python & raja 😄

[cameronrutherford]

@abhyshr I think this is good to merge if you are ok with how new README.md looks

[cameronrutherford]

I am going to merge this with one more approval from either @abhyshr or @ryandanehy

cc @pelesh if you also want to try the images out and give feedback on my instructions

[abhyshr]

Will it take any space in the ExaGO repository for the Cache?

[cameronrutherford]

Will it take any space in the ExaGO repository for the Cache?

So far I don't know if there is a limit on how much you can store in the cache. Since spack generates a hash for each unique package, and we force push every time to refresh things, we generally will only have copies of ExaGO that are duplicate and build up.

88 tracks a way to clean this out on a semi frequent basis, but you can always delete the whole thing whenever you want. A clean re-build takes about 1.5hrs total, and then sequential builds take <10 minutes.

[cameronrutherford]

@DJ-pnnl is there a limit on the number of packages we can have? Do you know of easy ways to clean things out that might be older than n-weeks?

[abhyshr]

Let me rephrase...do you need to store anything in the ExaGO repository on Github for the buildcache?

[cameronrutherford]

Let me rephrase...do you need to store anything in the ExaGO repository on Github for the buildcache?

The only code for this is in .github/workflows/spack_cpu_build.yml, along with some documentation.

The rest (the buildcache) is stored independently from the repo and doesn't change the size of the clone.

[cameronrutherford]

If @jaelynlitz or @ryandanehy can approve (I need a re-review because I rebased), then this will auto merge when the pipeline succeeds

[DJ-pnnl]

I found that there is a 1,000 package limit but I haven't found that in documentation. Currently, ExaGO is the only package this org has though. I'm still looking into adding cleanup to the packages as well.

Thanks, DJ

From: Cameron Rutherford @.> Sent: Thursday, November 30, 2023 6:52 PM To: pnnl/ExaGO @.> Cc: DJ Moore @.>; Mention @.> Subject: Re: [pnnl/ExaGO] Add Spack buildcache to CPU Ubuntu builds (PR #85)

Check twice before you click! This email originated from outside PNNL.

@DJ-pnnlhttps://github.com/DJ-pnnl is there a limit on the number of packages we can have? Do you know of easy ways to clean things out that might be older than n-weeks?

- Reply to this email directly, view it on GitHubhttps://github.com/pnnl/ExaGO/pull/85#issuecomment-1835371846, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AXARDEFGU4W6KR2RFH6YQRTYHFA6ZAVCNFSM6AAAAAA7VL7PLWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZVGM3TCOBUGY. You are receiving this because you were mentioned.Message ID: @.**@.>>