search

NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)
https://global-workflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
74 stars 165 forks source link

Re-enable C48mx500_3DVarAOWCDA test #2438

Closed WalterKolczynski-NOAA closed 4 months ago

WalterKolczynski-NOAA commented 5 months ago

What new functionality do you need?

The C48mx500_3DVarAOWCDA had to be disabled as the ocean prep obs job was failing when some categories of obs were not present in dump. Once GDASApp has been updated, the submodule in workflow should be updated and the test turned back.

What are the requirements for the new functionality?

C48mx500_3DVarAOWCDA completes successfully.

Acceptance Criteria

Suggest a solution (optional)

No response

JessicaMeixner-NOAA commented 5 months ago

FYI @guillaumevernieres @CatherineThomas-NOAA

WalterKolczynski-NOAA commented 5 months ago

This issue isn't resolved until the GDASapp hash has been updated in workflow

guillaumevernieres commented 5 months ago

I didn't mean to close it @WalterKolczynski-NOAA , sorry! We just merged a fix in the GDASApp.

JessicaMeixner-NOAA commented 4 months ago

This should be able to be re-enabled after https://github.com/NOAA-EMC/global-workflow/pull/2514 (updating GSI hash) and https://github.com/NOAA-EMC/global-workflow/pull/2420 (updating gdas app hash) are merged.

JessicaMeixner-NOAA commented 4 months ago

The GSI hash has been updated. We are now waiting on #2420 to update the GDAS app hash. Hopefully this will be done soon so we can get this test turned back on.

JessicaMeixner-NOAA commented 4 months ago

2420 is now merged, however there is a new issue https://github.com/NOAA-EMC/global-workflow/issues/2528 preventing this test from being turned back on.

JessicaMeixner-NOAA commented 4 months ago

The issue #2528 has been resolved enough to get past the first error. Now we have a new error that: FileNotFoundError: [Errno 2] No such file or directory: '/scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/iau_02/C48mx500_3DVarAOWCDA_t03/COMROOT/c48t03/gdas.20210324/12//model_data/ice/restart/20210324.180000.cice_model.res.nc' in gdasocnanalprep

JessicaMeixner-NOAA commented 4 months ago

Currently in COM for ice restarts I have: 

$ls /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/iau_02/C48mx500_3DVarAOWCDA_t03/COMROOT/c48t03/gdas.20210324/12/model_data/ice/restart/
20210324.150000.cice_model.res.nc

whereas before we had: 

$ ls /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/turnonmarine/c48ci01/COMROOT/c48t01/gdas.20210324/12/model_data/ice/restart
20210324.150000.cice_model.res.nc  20210324.180000.cice_model.res.nc  20210324.210000.cice_model.res.nc

I believe this is a change from https://github.com/NOAA-EMC/global-workflow/pull/2510 @aerorahul should additional ice restart files needed for DA be copied to COM or should the DA jobs be looking for this data in another directory?

guillaumevernieres commented 4 months ago

~If this was tested without the "IAU" flag on, then there's a bug in the g-w. The restarts should be for 18z, not 15z. Also, it looks like the atmos restarts are available at 15-18 and 21, but this isn't the case for mom6 and cice6.~

It's a bug in the marine DA @JessicaMeixner-NOAA . We're inserting the analysis in the wrong restart when DO_IAU is switched on ... Oops.