DavidHuber-NOAA commented 4 months ago

Description

1) Adds a lot of comments to the jinja templates for archiving 2) Rearranges the gdas and enkf templates to a more logical order 3) Fixes a couple of bugs in the enkf archiving of increments and analyses 4) Disables archiving for the half cycle 5) Removes the FITSARC key from config.base and arcdir.yaml.j2, instead relying on DO_FIT2OBS 6) Updates wxflow to add the option to not allow undefined variables when parsing jinja templates and invokes this feature when running archives

Resolves #2612

Type of change

Bug fix (fixes something broken)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

Cycled test on Hera

Checklist

[x] Any dependent changes have been merged and published
[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] My changes generate no new warnings
[ ] New and existing tests pass with my changes

AndrewEichmann-NOAA commented 4 months ago

A test run with hash 7d2c539f45194cd4e5b21bfd4b83a9480189cd0f (May 21) had arch fail on the first half-cycle with:

  File "/scratch1/NCEPDEV/da/Andrew.Eichmann/fv3gfs/develop/global-workflow/ush/python/pygfs/task/archive.py", line 199, in _create_fileset
    raise FileNotFoundError(f"FATAL ERROR: Required file, directory, or glob {item} not found!")
FileNotFoundError: FATAL ERROR: Required file, directory, or glob gdas.20210630/00/model_data/ocean/history/gdas.ocean.t00z.inst.f000.nc not found!

In the two subsequent cycles arch fails with the following:

  File "/scratch1/NCEPDEV/da/Andrew.Eichmann/fv3gfs/develop/global-workflow/ush/python/pygfs/task/archive.py", line 199, in _create_fileset
    raise FileNotFoundError(f"FATAL ERROR: Required file, directory, or glob {item} not found!")
FileNotFoundError: FATAL ERROR: Required file, directory, or glob gdas.20210630/06/analysis/ocean/gdas.t06z.ocn.adt_rads_all.stats.csv not found!

@guillaumevernieres tells me that the csv file in question is produced only on 00 cycles.

guillaumevernieres commented 4 months ago

@DavidHuber-NOAA , there's a bug in the archiving of the csv files. applying this patch should fix the issue:

diff --git a/parm/archive/gdasocean_analysis.yaml.j2 b/parm/archive/gdasocean_analysis.yaml.j2
index 0c43cd40..042f434e 100644
--- a/parm/archive/gdasocean_analysis.yaml.j2
+++ b/parm/archive/gdasocean_analysis.yaml.j2
@@ -9,7 +9,7 @@ gdasocean_analysis:
         {% for domain in ["ocn", "ice"] %}
         - '{{ COM_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}{{domain}}.bkgerr_stddev.nc'
         - '{{ COM_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}{{domain}}.incr.nc'
-        - '{{ COM_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}{{domain}}ana.nc'
+<        - '{{ COM_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}{{domain}}ana.nc'
         {% if NMEM_ENS > 2 %}
         - '{{ COM_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}{{domain}}.recentering_error.nc'
         {% endif %}
@@ -20,8 +20,6 @@ gdasocean_analysis:
         - '{{ COM_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}ocn.ssh_total_stddev.nc'
         - '{{ COM_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}ocn.steric_explained_variance.nc'
         {% endif %}
-        - '{{ COM_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}ocn.adt_rads_all.stats.csv'
-        - '{{ COM_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}ocn.icec_amsr2_north.stats.csv'
-        - '{{ COM_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}ocn.icec_amsr2_south.stats.csv'
+        - '{{ COM_OCEAN_ANALYSIS | relpath(ROTDIR) }}/*.stats.csv'
         - '{{ COM_OCEAN_ANALYSIS | relpath(ROTDIR) }}/diags/*.nc4'
         - '{{ COM_OCEAN_ANALYSIS | relpath(ROTDIR) }}/yaml/*.yaml'

DavidHuber-NOAA commented 4 months ago

Thanks for the heads up @AndrewEichmann-NOAA @guillaumevernieres. This PR will disable archiving for the first half cycle, so that should resolve the missing gdas.ocean.t00z.inst.f000.nc error.

For the csv files, I would like to test out the suggested change. Could you share your experiment setup so I can run a test case?

guillaumevernieres commented 4 months ago

Thanks for the heads up @AndrewEichmann-NOAA @guillaumevernieres. This PR will disable archiving for the first half cycle, so that should resolve the missing gdas.ocean.t00z.inst.f000.nc error.

For the csv files, I would like to test out the suggested change. Could you share your experiment setup so I can run a test case?

No problem and thanks for doing this work. This was discovered while running the c384/0.25, but that could be tested at lower res with the existing test, you'll just need to extend the date since it only does 1.5 cycle:

https://github.com/NOAA-EMC/global-workflow/blob/7d2c539f45194cd4e5b21bfd4b83a9480189cd0f/ci/cases/pr/C48mx500_3DVarAOWCDA.yaml#L14

guillaumevernieres commented 4 months ago

Oops ... 1 more thing @DavidHuber-NOAA , turns out we don't have enough marine obs in glopara to do enough cycles to trigger the archiving. You'll have to update DMPDIR: https://github.com/NOAA-EMC/global-workflow/blob/7d2c539f45194cd4e5b21bfd4b83a9480189cd0f/parm/config/gfs/yaml/defaults.yaml#L55

to

DMPDIR: /scratch1/NCEPDEV/da/common/

DavidHuber-NOAA commented 4 months ago

OK, thanks for the heads up @guillaumevernieres.

DavidHuber-NOAA commented 4 months ago

@guillaumevernieres Alright, I have a fix in for the csv files.