Use earthshot slurm account at PNNL (deception & newell)

[jaelynlitz]

Transition CI to use the earthshot account on PNNL HPC resources

Merge request type

• [ ] New feature
• [X] Resolves bug
• [ ] Documentation
• [ ] Other

Relates to

• [ ] OPFLOW
• [ ] SOPFLOW
• [ ] SCOPFLOW
• [ ] TCOPFLOW
• [ ] CMake build system
• [X] Spack configuration
• [ ] Manual
• [ ] Web docs
• [X] CI

This MR updates

• [ ] Header files
• [ ] Source code
• [ ] CMake build system
• [ ] Spack configuration
• [ ] Web docs
• [ ] Manual
• [ ] Other

[cameronrutherford]

I at least kicked off a new pipeline... Things are still failing and looks like we need to bugfix quite a few edge cases in our spack scripts that didn't present until now...

[cameronrutherford]

Okay I tried poking around, and some very confusing error messages.

• Newell was able to build, but now we are seeing errors running tests about various MPI/networking related services being unavailable. I am guessing a quick hack around to get MPI hello world working will highlight any issues
• Deception/Incline are both having the same issue building PETSc. For some reason:
  • Latest PETSc version in spack in 3.20.4. Latest released minor version is 3.20.9
  • PETSc is trying to execute mpiexec -n 1 --oversubscribe in the configure stage of building on both platforms, and both processes are timing out. You would assume that Newell would have the same behavior, but it doesn't

Debugging steps I have brainstormed:

• [ ] Try a different MPI / Compiler
• [ ] Try downgrading PETSc and reporting a PETSc bug
• [ ] Modifying SLURM arguments to correctly allocate threads/cores
• [ ] Running build process manually to reproduce

@jaelynlitz I will leave this until you get back...