search

NCAR / ccpp-framework

Common Community Physics Package (CCPP)
http://www.dtcenter.org/community-code/common-community-physics-package-ccpp/
Other
26 stars 63 forks source link

ccpp_prebuild.py: add support for chunked arrays #519

Closed climbfuji closed 5 months ago

climbfuji commented 7 months ago

Description

Add support for using chunked arrays to ccpp_prebuild.py while maintaining backward compatibility with the use of blocked data structures.

This PR adds support (and a test) for using chunked arrays as an alternative to blocked data structures to ccpp_prebuild.py. The idea behind this is that we can convert the ufs-weather-model to using contiguous arrays that are sent in chunks to the physics for (OpenMP-)parallel processing in the time integration (run) phase. The expectation is that this is at least as fast as the current implementation that uses blocked data structures (because the need to copy non-contiguous arrays into contiguous arrays and back for the init, timestep_init, timestep_finalize and finalize phases is removed). There is also evidence from extensive testing with CESM by CISL that sending chunks of contiguous arrays is faster.

Note. This PR does not yet make any changes for the ufs-weather-model, NEPTUNE or the SCM. It only provides the capability (and a test case that can serve as a how-to) for using contiguous arrays that can be sent to the physics in chunks in the run phase. There are limitations with how we (within ccpp_prebuild) treat inactive (unallocated) host model data that do not allow us to switch over to contiguous arrays immediately. I left several notes in the files changed in this PR to highlight the problems. We need to find a different way to handle inactive data as a next step so that we can make the switch to contiguous arrays afterwards. This will have the positive (!) side effect that it will address concerns raised by NCO that the current approach violates operational requirements.

User interface changes?: No

Issues

Fixes https://github.com/NCAR/ccpp-framework/issues/520

Testing

test removed: none test added: added test_prebuild/test_chunked_data/* unit/system tests: all capgen unit/system tests pass (test/run_test.sh); all ccpp_prebuild unit/system tests pass (everything in test_prebuild) end-to-end tests: full regression tests with ufs-weather-model on Hera with Intel and GNU against existing baseline; all tests pass; see also https://github.com/ufs-community/ufs-weather-model/pull/2066 for final regression testing manual testing: extensively tested on my macOS with apple-clang+gfortran

Dependencies / associated PRs

This PR is part of a set of PRs that need to be tested and merged together:

mkavulich commented 6 months ago

Waiting for RTs to succeed on https://github.com/ufs-community/ufs-weather-model/pull/2066

climbfuji commented 6 months ago

@dustinswales @grantfirl I think we can move ahead with this PR and the associated ccpp-physics PR. For that one, we wanted to wait for the outcome of our active attribute discussion. Now that we know that we would have to make changes anyway to the ozone physics, we might as well merge this here first (make it compliant with current CCPP rules) and then update ALL of the potentially inactive (= optional) variables in a next round?

zach1221 commented 5 months ago

@climbfuji testing is complete on ufs-wm PR #2066. Are you able to merge this ccpp-physics framework PR?

climbfuji commented 5 months ago

I can merge both but would prefer if @dustinswales or @grantfirl did that