search

NCAR / ccpp-framework

Common Community Physics Package (CCPP)
http://www.dtcenter.org/community-code/common-community-physics-package-ccpp/
Other
26 stars 64 forks source link

ccpp_track_variables.py is broken since CCPP physics directory structure was reorganized #580

Closed mkavulich closed 1 week ago

mkavulich commented 1 month ago

Description

Because of the change in directory structure for CCPP physics (https://github.com/ufs-community/ccpp-physics/pull/99), there are now .meta files at different levels in the directory tree. The ccpp_track_variables.py script needs the location of these .meta files as an input argument to the script, but the call to glob.glob in the script does not use the recursive=True argument, so even if the user passes in the argument -m './physics/physics/**/' (which should include all subdirectories), the call to glob.glob only searches one level. Our simple test case only has .meta files at a single directory level, so we never caught this issue.

Steps to Reproduce

  1. git clone git@github.com:ufs-community/ufs-weather-model --recursive
  2. cd ufs-weather-model/FV3/ccpp
  3. Try to run the ccpp_track_variables.py tool:

This command worked prior to this change, but no longer does:

framework/scripts/ccpp_track_variables.py -c=config/ccpp_prebuild_config.py  -s=suites/suite_FV3_RRFS_v1beta.xml -v air_temperature_of_new_state -m ./physics/physics/
Traceback (most recent call last):
  File "/Volumes/d1/kavulich/CCPP/v7_release/ufs-weather-model/FV3/ccpp/framework/scripts/ccpp_track_variables.py", line 217, in <module>
    track_variables(args.sdf,args.metadata_path,args.config,args.variable,args.debug)
  File "/Volumes/d1/kavulich/CCPP/v7_release/ufs-weather-model/FV3/ccpp/framework/scripts/ccpp_track_variables.py", line 202, in track_variables
    (success, var_graph) = create_var_graph(suite, variable, config, metadata_path, run_env)
  File "/Volumes/d1/kavulich/CCPP/v7_release/ufs-weather-model/FV3/ccpp/framework/scripts/ccpp_track_variables.py", line 100, in create_var_graph
    metadata_dict=create_metadata_filename_dict(metapath)
  File "/Volumes/d1/kavulich/CCPP/v7_release/ufs-weather-model/FV3/ccpp/framework/scripts/ccpp_track_variables.py", line 74, in create_metadata_filename_dict
    raise Exception(f'No files found in {metapath} with ".meta" extension')
Exception: No files found in ./physics/physics/ with ".meta" extension

This command should work with the new directory structure, but does not because of the problem mentioned above:

framework/scripts/ccpp_track_variables.py -c=config/ccpp_prebuild_config.py      -s=suites/suite_FV3_RRFS_v1beta.xml -v air_temperature_of_new_state -m './physics/physics/**/'
Traceback (most recent call last):
  File "/Volumes/d1/kavulich/CCPP/v7_release/ufs-weather-model/FV3/ccpp/framework/scripts/ccpp_track_variables.py", line 217, in <module>
    track_variables(args.sdf,args.metadata_path,args.config,args.variable,args.debug)
  File "/Volumes/d1/kavulich/CCPP/v7_release/ufs-weather-model/FV3/ccpp/framework/scripts/ccpp_track_variables.py", line 202, in track_variables
    (success, var_graph) = create_var_graph(suite, variable, config, metadata_path, run_env)
  File "/Volumes/d1/kavulich/CCPP/v7_release/ufs-weather-model/FV3/ccpp/framework/scripts/ccpp_track_variables.py", line 115, in create_var_graph
    raise Exception(f"Error, scheme '{scheme}' from suite '{suite.sdf_name}' "
Exception: Error, scheme 'GFS_time_vary_pre' from suite 'suites/suite_FV3_RRFS_v1beta.xml' not found in metadata files in ./physics/physics/**/

After the fix, this command works as expected:

framework/scripts/ccpp_track_variables.py -c=config/ccpp_prebuild_config.py      -s=suites/suite_FV3_RRFS_v1beta.xml -v air_temperature_of_new_state -m './physics/physics/**/'
For suite suites/suite_FV3_RRFS_v1beta.xml, the following schemes (in order for each group) use the variable air_temperature_of_new_state:
In group physics
  GFS_suite_stateout_reset_run (intent out)
  dcyc2t3_run (intent in)
  GFS_suite_stateout_update_run (intent out)
  get_phi_fv3_run (intent in)
  GFS_suite_interstitial_3_run (intent in)
  GFS_MP_generic_pre_run (intent in)
  mp_thompson_pre_run (intent in)
  mp_thompson_run (intent inout)
  mp_thompson_post_run (intent inout)
  GFS_MP_generic_post_run (intent in)
  maximum_hourly_diagnostics_run (intent in)
In group stochastics
  GFS_stochastics_run (intent inout)

Additional Context

None

Output

See above.

mkavulich commented 1 week ago

Fixed in #581