Update to v0.2.0

[xylar]

This merge brings over many of the changes from the compass PR https://github.com/MPAS-Dev/compass/pull/641. This includes:

• Bringing in a new mache with support for Intel on Perlmutter (not yet supported in polaris) and which allows for spack environment builds on non-E3SM machines
• Updates to a new pyremap that supports specifying the NetCDF format and (where appropriate) the xarray NetCDF engine
• Names spack clones the same as the spack environment that will be created in them (to avoid multiple developers trying to use the same clone of spack)
• Adds support for deploying on supported machines that aren't part of mache (non-E3SM machines), including documentation and an example (my linux laptop)
• Update versions of HDF5, NetCDF-C, NetCDF-Fortran, Parallel-NetCDF, ESMF and PETSc in spack environments
• Drops the --with_netlib_lapack flag, since we currently require that library always wehen PETSc is installed.

It also switches unknown machines to using the conda-forge versions of libpnetcdf and parallelio, which are newly available.

Checklist

• [x] Developer's Guide has been updated
• [x] Documentation has been built locally and changes look as expected
• [x] Testing comment in the PR documents testing used to verify the changes

Deployed

MPAS-Ocean with pr:

• [x] Anvil
  • [x] intel and impi
  • [x] intel and openmpi
  • [x] gnu and openmpi
  • [x] gnu and mvapich
• [x] Chicoma
  • [x] gnu and mpich
• [x] Chrysalis
  • [x] intel and openmpi
  • [x] gnu and openmpi
• [x] Compy
  • [x] intel and impi
• [x] Perlmutter
  • [x] gnu and mpich

[xylar]

I tried to do a spack build on my Mac laptop but I'm having some pretty fundamental trouble with bash on the Mac and wasn't able to get Spack to behave.

[xylar]

@altheaden, this worked for me on the Mac, at least up to the point of building MPAS-Ocean. I'm still doing that so haven't actually run tests yet.

Please let me know if it works for you. You'll want to check out this branch and run:

./configure_polaris_envs.py --recreate --verbose

and optionally additional options like --env_name and --conda if you need them.

Update: I was able to run most of the pr suite on my Mac (3 tests failed because I only have 4 cores).

[xylar]

Testing

I was able to run the pr suite on Perlmutter and on my Linux and Mac laptops (except for tests that are too big). I'll be testing on all machines as part of deploying the spack environment.

[altheaden]

@xylar I am able to make the environment when I use your branch, without issue. I'll build and run the PR suite now and see how everything looks on my end.

[altheaden]

@xylar I'm having some make errors, and I'm not sure if it's related to these issues or something specific to my mac. Here's what I'm seeing:

make clean && make gfortran-clang
...
mpas_io.F:280:41:

  280 |             pio_iotype = PIO_iotype_adios
      |                                         1
Error: Symbol ‘pio_iotype_adios’ at (1) has no IMPLICIT type; did you mean ‘pio_iotype’?
make[3]: *** [Makefile:126: mpas_io.o] Error 1
make[3]: Leaving directory '/Users/althea/polaris/main/e3sm_submodules/E3SM-Project/components/mpas-framework/src/framework'
make[2]: *** [Makefile:26: frame] Error 2
make[2]: Leaving directory '/Users/althea/polaris/main/e3sm_submodules/E3SM-Project/components/mpas-framework/src'
make[1]: *** [Makefile:1035: framework] Error 2
make[1]: Leaving directory '/Users/althea/polaris/main/e3sm_submodules/E3SM-Project/components/mpas-ocean'
make: *** [Makefile:323: gfortran-clang] Error 2

Prior to this I updated the submodules, and then I tried deleting and re-updating the submodule folder, and still no luck.

[xylar]

@altheaden, can you check to see if the following is in your load script?

export HAVE_ADIOS=false

Also, are you using the submodule from within the update-to-0.2.0 worktree or the one from main? If the latter, it might not be updated enough. (This is a special situation where I updated E3SM-Project in this PR so you can't use the one from main.)

[altheaden]

@xylar I was updating the ones in main, so I'll just update them all from this branch and make there. I do see that line in the load script for the branch! I will update you in a while after the submodules are up to date.

[altheaden]

@xylar I was able to build and run everything, and the only test that failed is the ocean/inertial_gravity_wave/convergence test. Do you want me to do any additional testing on the mac? I can send the log file for the inertial gravity test if that seems like it was supposed to pass - not sure if that's just a resources issue or what.

[xylar]

@altheaden, sounds fine. That's the test that failed for me on my laptop, too. It needs a minimum of 32 cores for the highest resolution forward step so it won't work on laptops in general. Nothing to worry about.

Thanks for testing!

[xylar]

@cbegeman, is this something you want to test (and have time to test)? Or would you be happy trusting my testing and @altheaden's?

[cbegeman]

@cbegeman, is this something you want to test (and have time to test)? Or would you be happy trusting my testing and @altheaden's?

I'm happy to trust your and @altheaden's testing. I don't have time to contribute to testing this week.

[altheaden]

Since the LCRC servers are back online now, I will continue some testing on chrysalis and anvil.

[xylar]

@altheaden, I'm just waiting for your approval to merge. If you have already done LCRC testing, great! Otherwise, your Mac testing is plenty.

[altheaden]

I tested on Chrysalis with intel and openmpi and everything worked as expected.