search

NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)
https://global-workflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
70 stars 162 forks source link

Convert staging job to python for forecast-only mode #2651

Open KateFriedman-NOAA opened 4 weeks ago

KateFriedman-NOAA commented 4 weeks ago

Description

This PR converts the staging job from shell to python. The existing forecast-only stage_ic is converted to use python and yaml via jinja parsing. This PR does not impact the existing ROTDIR symlink population done for cycled mode via the setup scripts.

Changes in this PR:

  1. Rename scripts/exglobal_stage_ic.sh to scripts/exglobal_stage_ic.py.
  2. Update jobs/JGLOBAL_STAGE_IC to use .py script extension. Move GDATE/gPDY/gcyc settings up to JJOB from ex-script, as well as move up COM* variable declarations and MEMDIR[_ARRAY] settings.
  3. Add PYTHONPATH export to jobs/rocoto/stage_ic.sh.
  4. Create parm/stage folder to hold newly created yamls (*.yaml.j2) for each initial condition set currently handled in existing scripts/exglobal_stage_ic.sh. Included yamls for warm start (fv3_warm.yaml.j2) and DO_NEST=YES (fv3_nest.yaml.j2) to retain functionality in existing scripts/exglobal_stage_ic.sh but have not tested these configurations in forecast-only mode.
  5. Create ush/python/pygfs/task/stage.py to house staging job python functions for call from scripts/exglobal_stage_ic.py.
  6. Add export USE_OCN_PERTURB_FILES=".false." to gfs config.base; need variable as key in staging job python regardless of RUN.
  7. Remove stage_ic job rocoto dependencies from xml. Do not need and removes area of duplicate maintenance.

Follow-up PRs will add stage_ic job to cycled mode and extended capabilities as needed as part of issue #2475.

Resolves #2650

Type of change

Change characteristics

How has this been tested?

Ran the forecast-only CI tests on Hera:

Outputs match CI tests run from develop. Logs available for review.

Snippet from C48 S2SWA GEFS stage_ic job log showing the mkdir and copy of mem002 files into ROTDIR:

2024-06-03 13:50:27,622 - INFO     - file_utils  : Created /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/atmos/input
2024-06-03 13:50:27,628 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/atmos/gfs_ctrl.nc to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/atmos/input
2024-06-03 13:50:27,734 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/atmos/gfs_data.tile1.nc to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/atmos/input
2024-06-03 13:50:27,879 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/atmos/gfs_data.tile2.nc to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/atmos/input
2024-06-03 13:50:27,983 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/atmos/gfs_data.tile3.nc to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/atmos/input
2024-06-03 13:50:28,174 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/atmos/gfs_data.tile4.nc to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/atmos/input
2024-06-03 13:50:28,317 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/atmos/gfs_data.tile5.nc to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/atmos/input
2024-06-03 13:50:28,463 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/atmos/gfs_data.tile6.nc to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/atmos/input
2024-06-03 13:50:28,544 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/atmos/sfc_data.tile1.nc to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/atmos/input
2024-06-03 13:50:28,583 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/atmos/sfc_data.tile2.nc to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/atmos/input
2024-06-03 13:50:28,614 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/atmos/sfc_data.tile3.nc to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/atmos/input
2024-06-03 13:50:28,694 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/atmos/sfc_data.tile4.nc to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/atmos/input
2024-06-03 13:50:28,721 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/atmos/sfc_data.tile5.nc to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/atmos/input
2024-06-03 13:50:28,932 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/atmos/sfc_data.tile6.nc to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/atmos/input
2024-06-03 13:50:28,941 - INFO     - file_utils  : Created /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/wave/restart
2024-06-03 13:50:29,101 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/wave/20210323.120000.restart.glo_500 to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/12/mem002/model_data/wave/restart
2024-06-03 13:50:29,206 - INFO     - file_utils  : Created /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/06/mem002/model_data/ocean/restart
2024-06-03 13:50:29,478 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/ocean/20210323.120000.MOM.res.nc to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/06/mem002/model_data/ocean/restart
2024-06-03 13:50:29,485 - INFO     - file_utils  : Created /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/06/mem002/model_data/ice/restart
2024-06-03 13:50:29,526 - INFO     - file_utils  : Copied /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/gefs_test/2021032312/mem002/ice/20210323.120000.cice_model.res.nc to /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/teststage_C48_S2SWA_gefs_c/gefs.20210323/06/mem002/model_data/ice/restart
2024-06-03 13:50:29,621 - INFO     - stage       :   END: pygfs.task.stage.execute_stage
2024-06-03 13:50:29,622 - DEBUG    - stage       :  returning: None
2024-06-03 13:50:29,622 - INFO     - root        :   END: __main__.main
2024-06-03 13:50:29,622 - DEBUG    - root        :  returning: None
+ JGLOBAL_STAGE_IC[67]: [[ 0 -ne 0 ]]
+ JGLOBAL_STAGE_IC[75]: cd /scratch1/NCEPDEV/stmp2/Kate.Friedman/RUNDIRS/teststage_C48_S2SWA_gefs_c
+ JGLOBAL_STAGE_IC[76]: [[ NO = \N\O ]]
+ JGLOBAL_STAGE_IC[76]: rm -rf /scratch1/NCEPDEV/stmp2/Kate.Friedman/RUNDIRS/teststage_C48_S2SWA_gefs_c/stage_ic.2811565
+ JGLOBAL_STAGE_IC[78]: exit 0
+ JGLOBAL_STAGE_IC[1]: postamble JGLOBAL_STAGE_IC 1717422601 0
+ preamble.sh[70]: set +x
End JGLOBAL_STAGE_IC at 13:50:30 with error code 0 (time elapsed: 00:00:29)
+ stage_ic.sh[20]: status=0
+ stage_ic.sh[23]: exit 0
+ stage_ic.sh[1]: postamble stage_ic.sh 1717422598 0
+ preamble.sh[70]: set +x
End stage_ic.sh at 13:50:30 with error code 0 (time elapsed: 00:00:32)
_______________________________________________________________
Start Epilog on node h22c03 for job 61225255 :: Mon Jun  3 13:50:30 UTC 2024
KateFriedman-NOAA commented 4 weeks ago

Got some good feedback from reviewers (offline and online). Will ponder feedback and work on updates. Will leave this draft PR open while working on it. Thank you all so far!

KateFriedman-NOAA commented 2 weeks ago

Will be making some iterative commits. Please do not rereview until rerequested.