search

E3SM-Project / scream

Fork of E3SM used to develop exascale global atmosphere model written in C++
https://e3sm-project.github.io/scream/
Other
79 stars 56 forks source link

P3 lookup text file being read by all MPI ranks #3023

Open ndkeen opened 1 month ago

ndkeen commented 1 month ago

I don't think we should be letting all MPI rank try to read the same P3 lookup text file as it can cause some issues on the filesystems esp at scale (and slow us down).

I implemented a read-on-rank0, broadcast solution to test for E3SM components/eam/src/physics/p3/eam/micro_p3.F90, but having trouble in scream components/eam/src/physics/p3/scream/micro_p3.F90 as I don't have access to same modules. So I wanted to test it was ok before making issue, but might need more attention.

For example, don't immediately have access to:

   use mpishorthand, only: mpir8
   use spmd_utils,   only: mpicom

Perhaps someone has already thought of a better solution anyway?

ambrad commented 1 month ago

This approach might work:

Move this line: https://github.com/E3SM-Project/scream/blob/5ad00c2afaf5ca20a6fd6d8584a6671cd7dcefe0/components/eamxx/src/physics/p3/eamxx_p3_process_interface.cpp#L252 down to just above here: https://github.com/E3SM-Project/scream/blob/5ad00c2afaf5ca20a6fd6d8584a6671cd7dcefe0/components/eamxx/src/physics/p3/eamxx_p3_process_interface.cpp#L423 Then call that whole block only on the master rank. Finally, broadcast the table data to the other ranks.

However, the approach will fail if there are C++-affecting side effects in p3_init.