Work Checklist - Prototyping a MPAS-A v8 OpenACC Port

[gdicker1]

This issue is meant to collect, discuss, and track efforts to create a new prototype branch of the MPAS-Model.

It is desired that this branch:

• is based on some version of the v8.x.x MPAS-Dev/MPAS-Model code
• utilizes heterogeneous computing systems (CPU+GPU), at least through OpenACC directives and other modifications
• focuses on the general, shared MPAS framework such that it remains interoperable with (and non-specific to) any core
• (eventually) focuses on the "dynamics" of the atmosphere core such that:
  • various stand-alone configurations are possible (e.g. kessler, convection_permitting, or none for physics schemes)
  • the atmosphere core can be easily used as a "dycore" in various community earth system models (e.g. CAM, CAM-SIMA, EarthWorks)

Current work items:

• [x] Expand initial work
  • [x] edit the settings of my (this) fork of MPAS-Model for group collaboration
  • [x] create branch based on specific version of MPAS-Dev/MPAS-Model
  • [x] enlist/entrain collaborator(s)
    • invite collaborators on my fork of MPAS-Model
    • tag them in this discussion
  • [ ] other...
• [x] Given the branch based on specific MPAS-Model "version", add a minimal ("hello, world" level) OpenACC test
  • See MPAS-Dev/MPAS-Model #1175

Incremental Steps

A.k.a. Roadmap

1. • [x] (Minimal) Build with OpenACC enabled
2. • [x] Verify ability to run on GPU -> "minimal OpenACC test" for 1 MPI rank
3. • [ ] Identify crucial infrastructure to be ported using OpenACC or created
     1. • [ ] Add routine to map MPI ranks to GPU devices
4. • [ ] Begin incrementally porting and creating infrastructure routines until "critical point"[^1]
5. • [ ] Identify crucial routines in core_atmosphere, driver, and elsewhere to port using OpenACC
6. • [ ] Begin incrementally porting routines until "critical point" using select test cases [^2]
     1. • [x] Port atm_compute_solve_diagnostics_work
     2. • [x] Port atm_advance_scalars_work
     3. • [x] Port atm_advance_scalars_mono_work
        • Michael accomplished these in MPAS-Dev/MPAS-Model #1176.
     4. • [ ] Port atm_compute_solve_diagnostics
     5. • [ ] Port remaining routines in dycore...
7. • [ ] Make improvements on the port to at least ensure "robustness" [^3]
8. • [ ] [END OF PROJECT] Assess next steps and how to integrate this to other projects (e.g. EarthWorks, MPAS-Dev/MPAS-Model repo)

[^1]: Here "critical point" is to mean some point that is "sufficient" for future work [^2]: NOTE: the test cases are TBD and may also be an incremental set of tests [^3]: NOTE: this means addressing memory management and maybe any performance issues that need to be addressed.

[gdicker1]

• [ ] enlist/entrain collaborator(s)

  • invite collaborators on my fork of MPAS-Model
  • tag them in this discussion

@mgduda, I think this comment satisfies my first check box :) Let me know what you think so far!

[gdicker1]

• is based on some version of the v8.x.x MPAS-Dev/MPAS-Model code

This has already been discussed some. The thought is to use "the current version of MPAS-A that is being used in ESCOMP/CAM's cam_development branch"

• Tag in ESCOMP/CAM that is the current head of cam_development: cam6_3_150
• Hash for [mpas] within Externals_CAM.cfg: b8c33daa

[mgduda]

• is based on some version of the v8.x.x MPAS-Dev/MPAS-Model code

This has already been discussed some. The thought is to use "the current version of MPAS-A that is being used in ESCOMP/CAM's cam_development branch"

* Tag in ESCOMP/CAM that is the current head of cam_development: [cam6_3_150](https://github.com/ESCOMP/CAM/tree/cam6_3_150)

* Hash for `[mpas]` within Externals_CAM.cfg: [b8c33daa](https://github.com/ESCOMP/CAM/blob/f9b91b90d1a085bcd99f1df005569954c4cf036b/Externals_CAM.cfg#L71C8-L71C16)

I think we should create the branch based on b8c33daa in this case.

[mgduda]

To restate the goal of this work as I see it, we want an OpenACC port of the MPAS-A dycore -- starting from b8c33daa -- that works in stand-alone MPAS-A as well as in CAM-SIMA. Along the way, we'll very likely need to do some minor infrastructure development, and we should also implement some tests of that infrastructure. All infrastructure tests (along with the initial "Is OpenACC working at a basic level" test) will go in the test core.

@gdicker1 Can you expand the list of incremental developments that might get us to this end goal? This is of course a first cut, and we'll refine it over time.

[gdicker1]

@gdicker1 Can you expand the list of incremental developments that might get us to this end goal? ...

@mgduda what do you think of the new "Incremental Steps" heading so far?

[mgduda]

The "Incremental Steps" are looking good. For (2), I think we could specify at that point we are only concerned with 1 MPI rank. Then, there could be a step (2.5) (or just make it (3) and shift the numbering of later steps) to add a routine to map MPI ranks to ACC devices. This routine could initially just apply a simple mapping (e.g., mod(rank, num_devices)), but would provide a place where more sophisticated logic -- like that in atm_component_role could be implemented later as needed. This rank-mapping routine should probably be placed in shared code, though, so it could be tested by the test core or so that it could ensure a conflict-free mapping for different MPAS components.

[mgduda]

As an idea for some sub-tasks under (6), we could:

• [ ] Port atm_compute_solve_diagnostics_work (nice because it isn't unwieldy and it contains no MPI communication)
• [ ] Port atm_advance_scalars_work (to see whether the strategy taken for atm_compute_solve_diagnostics_work applies to other dynamics routines
• [ ] Port atm_advance_scalars_mono_work (initially just do D->H transfers, halo exchanges, then H->D; but after computation in the routine has been ported, switch back to infrastructure and port halo exchange routines)
• [ ] Port remaining routines in dycore

[gdicker1]

• [x] create branch based on specific version of MPAS-Dev/MPAS-Model

The development branch for this work will be in: gdicker1/MPAS-Model:v8_oacc/develop. Features will be merged into it over time - especially by Pull Request and Review for OpenACC features.

[gdicker1]

The development branch for this work will be in: gdicker1/MPAS-Model:v8_oacc/develop. Features will be merged into it over time - especially by Pull Request and Review for OpenACC features.

Note: since my last update some OpenACC work has been merged into the main MPAS-Dev/MPAS-Model:develop branch. I will use that (or at least where PR #1175 has been merged).