Missing CICE hist files when cycle is not t00z and FHOUT_GFS_ICE is 24

[EricSinsky-NOAA]

What is wrong?

For cases where FHOUT_GFS_ICE is 24 and initialized at a time that is not 00Z, CICE files are not properly linked to COMROOT. This results in missing CICE hist files in COMROOT. After investigating this issue on WCOSS2, this happens because the raw CICE output is not averaged for a full 24 hours for the first lead time. For example, if the initialization time is 12Z and FHOUT_GFS_ICE is 24, the first lead time will only be averaged from 12Z to 00Z. Subsequent lead times are averaged from 00Z to 00Z. The global-workflow predetermines that the CICE output will be averaged from 12Z to 12z, which results in the global workflow creating links that do not match the raw CICE filenames, resulting in the CICE output not to save in COMROOT.

What should have happened?

It may be ideal that CICE be averaged for a full 24 hours for the first lead time. For example, if the initialization time is 12Z and FHOUT_GFS_ICE is 24, the first lead time should be averaged from 12Z to 12Z. Subsequent lead times should be averaged from 12Z to 12Z. An alternative positive outcome would be for the global-workflow to correctly predetermine the name of the CICE output in these particular cases.

What machines are impacted?

WCOSS2

Steps to reproduce

1. Set FHOUT_GFS_ICE to 24 in config.base.
2. Run a case that is not initialized at t00Z (e.g. t12z).

Additional information

This issue has been tested on WCOSS2, but it most likely impacts all platforms. The list of CICE output below illustrates how the global-workflow is not picking up the raw CICE output.

Do you have a proposed solution?

This issue has been initially addressed in PR #2561. In this PR, CICE_predet and CICE_postdet were modified so that the global-workflow can correctly predetermine the name of the raw CICE output (as illustrated in the image below). Also, the oceanice prod task was modified accordingly to account for the CICE filename changes in these cases. However, further fixes will need to be made to consider cases where FHMIN is not 0.

[aerorahul]

Can the output be 6 hrs, and a post-processing step be applied to do the averaging for 12hrs or 24hrs? Since the CICE model cannot do 24hr averaging if the starting time is not 00z for the first interval, the first 24h output as labeled by the workflow is not correct. The earliest 24h product that will be generated will be for the next period of 24hrs. Also, the last 24h period will be absent as the end time of the forecast will not be at 00z.

[EricSinsky-NOAA]

Can the output be 6 hrs, and a post-processing step be applied to do the averaging for 12hrs or 24hrs?

Since we can assign an FHOUT specifically for CICE now using FHOUT_ICE, we can keep FHOUT_ICE set to 6 (and perhaps warn users somewhere not to try FHOUT_ICE=24). I think it is a good idea to apply a separate post-processing step to perform an average of 12 hours or 24 hours. This would remove the need for the check_ice_netcdf.sh script and we can use the standard <datadep age=120> as a dependency, which would resolve issue #2721. The question is where to put this post-processing step.

Since the CICE model cannot do 24hr averaging if the starting time is not 00z for the first interval, the first 24h output as labeled by the workflow is not correct.

A fix has been made in the workflow so that the first filename is labelled as 12h instead of 24h in cases where cycle=t12z and FHOUT_ICE=24. Please see screenshot in the proposed solution section above.

Also, the last 24h period will be absent as the end time of the forecast will not be at 00z.

That is correct. The last lead time that is generated for CICE when FHMAX=120 is averaged from 84h-108h.