gfsmetp grid2grid verification files not generated

[RussTreadon-NOAA]

Description grid2grid verification files are not generated on the first submission of gfsmetpg2g1. A later rocotorewind and rocotoboot of gfsmetpg2g1 generated the expected grid2grid verification files.

Investigation of the gfsmetpg2g1 job log file found that verification files were not generated because expected gfs pgb files were not found in the online archive, $ARCDIR. The gfsarch job copies pgb*grib2 files from $ROTDIR to $ARCDIR. Thus, the gfsmetp suite of jobs should not be submitted before gfsarch completes. However, the rocoto xml only has gfspost as a dependency for gfsmetp.

                <dependency>
                        <and>
                                <metataskdep metatask="gfspost"/>
                        </and>
                </dependency>

Replacing the gfspost metatask dependency with a gfsarch task dependency would ensure gfsmetp jobs do not run before required data is in $ARCDIR.

There is another, less robust, way to address this. g-w config.metp contains variable $VRFYBACK_HRS. This variable is currently set to 0.

## DATE SETTINGS
export VRFYBACK_HRS="0"

Setting this variable to 24 would result in the CDATE execution of gfsmetp processing files for CDATE-24. While not guaranteed, it is more likely that the CDATE-24 gfsarch job is completed before the CDATE gfspost metatask is done.

Of the two options, adding a gfsarch dependency seems preferable. This issue is opened to bring this problem to the g-w and verification teams attention. What's the recommended solution from their expert perspective?

This issue is opened as a general and not bug issue since it's not clear if the reported issue a truly a bug, a user configuration error, or something else.

Requirements The default g-w configuration of gfsmetp should successfully generate all expected output.

[RussTreadon-NOAA]

tagging @malloryprow for awareness

[malloryprow]

Thanks @RussTreadon-NOAA. We had some prevous that removed the gfsarch dependency (https://github.com/NOAA-EMC/global-workflow/issues/907, https://github.com/NOAA-EMC/global-workflow/issues/437).

Thinking it out by typing here...VRFYBACK_HRS="24" should work. As the gfsarch step would have previously run. This is also what VSDB use to do, verify back 24 hours from whatever the CDATE is.

[RussTreadon-NOAA]

@malloryprow , I checked config files in my old EXPDIRS. Most set export VRFYBACK_HRS="24" in config.metp.

This approach assumes that the CDATE-24 gfsarch has completed before CDATE gfsmetp begins. This is usually the case but it's not guaranteed to be true. Adding a gfsarch dependency on gfsmetp ensures the archive job has copied files to $ARCDIR before gfsmetp starts.

Another option would be for gfsmetp to pick up gfs pgb grib2 files from $ROTDIR instead of $ARCDIR. In this case the existing gfsmetp dependency on metatask gfspost is sufficient.

If we want to go with the VRFYBACK_HRS="24" approach, we should replace VRFYBACK_HRS="0" with VRFYBACK_HRS="24" in g-w parm/config/gfs/config.metp and commit this to g-w develop.

[malloryprow]

Hmmm I did think I added that in https://github.com/NOAA-EMC/EMC_verif-global/blob/develop/ush/run_verif_global_in_global_workflow.sh. It was kind of hastily done.

[WalterKolczynski-NOAA]

Worth noting workflow is still using an orphaned branch of verify where we removed the module loads (feature_rm_gw_module_loading). AFAIK, that never got back into the verify develop because we've been waiting for a full working stack instead of my hacky one. So any changes that have been done to develop wouldn't be reflected.

[WalterKolczynski-NOAA]

Will resolve by making the dependency change this sprint.

[malloryprow]

Let me know if you need anything from me. We're in the last month of code finalization before code delivery Sept. 15th for EVS, but I can help out when and where I can.