Add initial version of ccpp_track_variables.py

mkavulich commented 2 years ago

Initial version of script for tracking which variables are modified by which schemes in a given suite.

This script takes in as input the following arguments:

A suite definition file
A path to scheme metadata files
A CCPP config file
A variable name

The script output changes based on the input:

If the given variable is found in any of the schemes for the given SDF, outputs a list of schemes (in their calling order, including duplicates) along with the variable intent for that scheme
If the given variable is not found, but a partial match is made to any of the found variables, the script outputs a list of schemes that contain variables with partial matches (along with those partial-match variables)
If the given variable is not found with no partial matches either, the script exits with an error indicating as such

This is a brand new script, so it does not change any existing interfaces. It does add a new variable and method to the Suite class: call_tree is a list of schemes in order (including duplicates) for the given Suite. The method make_call_tree is used to populate the call_tree list; it is not called by the write() method so must be called separately if desired.

Please provide feedback on the format and structure of the program!

I started with roughly the ccpp_prebuild script as a guide, but I am not tied to any particular style or philosophy here.

Testing: test removed: None unit tests: None system tests: None manual testing: Ran script with a variety of inputs, gives expected output. Examples below:

Running examples

No new python packages or other prerequisites are needed, so you can run on any platform that CCPP can already run on. It is also not necessary to build any code or run prebuild scripts; all you need is the ccpp-framework and ccpp-physics repositories, a model config file (this requirement can hopefully be removed in the future), and the xmls for the suites you wish to analyze.

Here is an example setup based on the ufs-weather-model on Hera:

git clone --recurse-submodules https://github.com/ufs-community/ufs-weather-model
cd ufs-weather-model/FV3/ccpp/framework/
git remote add mkavulich git@github.com:mkavulich/ccpp-framework
git fetch mkavulich
git checkout feature/track_variables_through_suite
cd ..

From here, you can run the following example commands:

$ framework/scripts/ccpp_track_variables.py --help
usage: ccpp_track_variables.py [-h] -s SDF -m METADATA_PATH -c CONFIG -v VARIABLE [--debug]

optional arguments:
  -h, --help            show this help message and exit
  -s SDF, --sdf SDF     suite definition file to parse
  -m METADATA_PATH, --metadata_path METADATA_PATH
                        path to CCPP scheme metadata files
  -c CONFIG, --config CONFIG
                        path to CCPP prebuild configuration file
  -v VARIABLE, --variable VARIABLE
                        variable to track through CCPP suite
  --debug               enable debugging output

Successful output: prints list of schemes that use the specified variable, along with the variable's intent

$ framework/scripts/ccpp_track_variables.py --config=config/ccpp_prebuild_config.py -s=suites/suite_FV3_GFS_v16_noahmp.xml -v canopy_water_amount -m ./physics/physics/
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_gas_optics_rrtmgp
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_optical_props
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_optical_props_arry
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_optical_props_1scl
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_optical_props_2str
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_optical_props_nstr
For suite suites/suite_FV3_GFS_v16_noahmp.xml, the following schemes (in order) use the variable canopy_water_amount:
GFS_phys_time_vary_init (intent in)
noahmpdrv_run (intent inout)
noahmpdrv_run (intent inout)

Unknown variable: script exits with descriptive error message

$ framework/scripts/ccpp_track_variables.py --config=config/ccpp_prebuild_config.py -s=suites/suite_FV3_GFS_v16_noahmp.xml -v volcano -m ./physics/physics/
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_gas_optics_rrtmgp
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_optical_props
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_optical_props_arry
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_optical_props_1scl
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_optical_props_2str
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_optical_props_nstr
ERROR: Variable volcano not found in any suites for sdf suites/suite_FV3_GFS_v16_noahmp.xml

Partial match for variable: outputs list of partial matches for each scheme

$ framework/scripts/ccpp_track_variables.py --config=config/ccpp_prebuild_config.py -s=suites/suite_FV3_GFS_v16_noahmp.xml -v latent_heat -m ./physics/physics/
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_gas_optics_rrtmgp
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_optical_props
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_optical_props_arry
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_optical_props_1scl
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_optical_props_2str
WARNING: Encountered closing statement "end type" without type name; assume type_name is ty_optical_props_nstr
ERROR: Variable latent_heat not found in any suites for sdf suites/suite_FV3_GFS_v16_noahmp.xml

Did find partial matches that may be of interest:

In GFS_suite_interstitial_2_run found variable(s) ['latent_heat_of_vaporization_of_water_at_0c']
In GFS_surface_generic_post_run found variable(s) ['surface_upward_potential_latent_heat_flux', 'soil_upward_latent_heat_flux', 'canopy_upward_latent_heat_flux', 'snow_deposition_sublimation_upward_latent_heat_flux', 'snow_freezing_rain_upward_latent_heat_flux', 'cumulative_soil_upward_latent_heat_flux_multiplied_by_timestep', 'cumulative_canopy_upward_latent_heat_flu_multiplied_by_timestep', 'cumulative_snow_deposition_sublimation_upward_latent_heat_flux_multiplied_by_timestep', 'cumulative_snow_freezing_rain_upward_latent_heat_flux_multiplied_by_timestep', 'cumulative_surface_upward_potential_latent_heat_flux_multiplied_by_timestep', 'multiplicative_tuning_parameter_for_reduced_latent_heat_flux_due_to_canopy_heat_storage']
In GFS_surface_composites_pre_run found variable(s) ['surface_upward_potential_latent_heat_flux_over_ice']
In GFS_surface_composites_post_run found variable(s) ['surface_upward_potential_latent_heat_flux', 'surface_upward_potential_latent_heat_flux_over_water', 'surface_upward_potential_latent_heat_flux_over_land', 'surface_upward_potential_latent_heat_flux_over_ice', 'kinematic_surface_upward_latent_heat_flux_over_water', 'kinematic_surface_upward_latent_heat_flux_over_land', 'kinematic_surface_upward_latent_heat_flux_over_ice', 'multiplicative_tuning_parameter_for_reduced_latent_heat_flux_due_to_canopy_heat_storage']
In sfc_nst_run found variable(s) ['latent_heat_of_vaporization_of_water_at_0c', 'latent_heat_of_fusion_of_water_at_0c', 'kinematic_surface_upward_latent_heat_flux_over_water', 'surface_upward_potential_latent_heat_flux_over_water']
In noahmpdrv_run found variable(s) ['latent_heat_of_vaporization_of_water_at_0c', 'latent_heat_of_fusion_of_water_at_0c', 'kinematic_surface_upward_latent_heat_flux_over_land', 'surface_upward_potential_latent_heat_flux_over_land', 'soil_upward_latent_heat_flux', 'canopy_upward_latent_heat_flux', 'snow_deposition_sublimation_upward_latent_heat_flux', 'snow_freezing_rain_upward_latent_heat_flux']
In sfc_sice_run found variable(s) ['latent_heat_of_vaporization_of_water_at_0c', 'surface_upward_potential_latent_heat_flux_over_ice', 'kinematic_surface_upward_latent_heat_flux_over_ice', 'kinematic_surface_upward_latent_heat_flux_over_water']
In GFS_PBL_generic_post_run found variable(s) ['instantaneous_surface_upward_latent_heat_flux', 'cumulative_surface_upward_latent_heat_flux_for_coupling_multiplied_by_timestep', 'surface_upward_latent_heat_flux_for_coupling', 'cumulative_surface_upward_latent_heat_flux_for_diag_multiplied_by_timestep', 'instantaneous_surface_upward_latent_heat_flux_for_diag', 'latent_heat_of_vaporization_of_water_at_0c', 'surface_upward_latent_heat_flux_from_coupled_process', 'kinematic_surface_upward_latent_heat_flux_over_water']
In satmedmfvdifq_run found variable(s) ['latent_heat_of_vaporization_of_water_at_0c', 'latent_heat_of_fusion_of_water_at_0c', 'instantaneous_surface_upward_latent_heat_flux']
In samfdeepcnv_run found variable(s) ['latent_heat_of_vaporization_of_water_at_0c']
In samfshalcnv_run found variable(s) ['latent_heat_of_vaporization_of_water_at_0c']

ligiabernardet commented 2 years ago

Thank you for this PR. This is a very helpful new addition to the CCPP. Going forward, we would like to have a visualization capability but having the script is the foundational step. Do we have any mechanism to make sure ccpp_track_variables does not break when new development is submitted?

mkavulich commented 2 years ago

@ligiabernardet I haven't included any unit or regression tests yet, but I can make that a priority before opening this PR for review. I'll need to familiarize myself with the existing testing framework first

mkavulich commented 2 years ago

After addressing some initial concerns (still a few to go) and updating for the most recent commits on main, this PR is ready for a proper review.

There are a few known issues/limitations right now, I think they can be addressed later on but I figured I'd point them out off the bat:

The extraneous statements of WARNING: Encountered closing statement "end type" without type name come from the metadata parser, and I believe are due to problems with the metadata itself.
The script can only be run from above the top-level framework directory in the ufs-weather-model examples as shown above; I believe this is a limitation specific to the ufs-weather-model ccpp_prebuild_config.py file and hopefully can be eliminated in the future when the config file is no longer needed.

mkavulich commented 2 years ago

@christinaholtNOAA I believe I have resolved or responded to all your comments, please let me know if you have any further comments or questions.

climbfuji commented 2 years ago

My main comments have been addressed, so I pass the baton to @gold2718 ...

grantfirl commented 2 years ago

@mkavulich Would you please update this to the latest main in anticipation of merging?

NCAR / ccpp-framework