Fix the model inconsistencies between 6 and 54 rank runs - Githubissues

ai2cm / pace

Implementation of the FV3GFS / SHiELD atmospheric model in Python

https://ai2cm.github.io/pace/

Apache License 2.0

37 stars 12 forks source link

Fix the model inconsistencies between 6 and 54 rank runs #283

Closed twicki closed 2 years ago

twicki commented 2 years ago

Purpose

We have seen inconsistencies between 6 and 54 rank runs. These should not eixist and get solved by this.

Code changes:

Add the allreduce method to Comm, MPIComm and NullComm
Assert LocalComm in allreduce because it requires synchronicity.
Removed the try-except block in generation.py in the _reduce_global_area_minmaxes function.
updated gt4py_utils.py to properly handle scalars

Checklist

Before submitting this PR, please make sure:

[x] You have followed the coding standards guidelines established at Code Review Checklist.
[x] Docstrings and type hints are added to new and updated routines, as appropriate
[x] For each public change and fix in pace-util, HISTORY has been updated

mcgibbon commented 2 years ago

When you merge can you update the PR title to describe what was fixed, so the merge commit says what was changed?

twicki commented 2 years ago

good point, done