Expand ctests to include additional obs for more realistic testing

[SamuelDegelia-NOAA]

As mentioned in our last meeting, we do not plan to add additional ctests for each new observation type. Instead, we will expand current ctests for each DA method (3DEnVar, GETKF, etc.) to use additional observations as the yamls get added to RDASApp. This will allow us to build up ctests for more realistic scenarios and observation counts.

My initial thoughts about how to implement this are:

• Generate new versions of the rrfs-test/testinput/*yaml files, combining the DA yamls in rrfs-test/validated_yamls/templates/basic_config with observation yamls in rrfs-test/validated_yamls/templates/obtype_config. This will make use of the gen_yaml.sh tool that @delippi has developed.
• The older yamls in rrfs-test/testinput are still useful for running smaller experiments, so I plan to move those into something like rrfs-test/expr_input.

[ShunLiu-NOAA]

@SamuelDegelia-NOAA This is a good way to move forward. @delippi will work on generating an unified ioda data set and yaml files for the two test cases. You two can discuss how to coordinate this effort.

[SamuelDegelia-NOAA]

@SamuelDegelia-NOAA This is a good way to move forward. @delippi will work on generating an unified ioda data set and yaml files for the two test cases. You two can discuss how to coordinate this effort.

That will be very helpful. @delippi would you be able to generate yaml files like those in rrfs-test/testinput but using the full set of observations as generated by gen_yaml.sh? I could also do this if you want to work on the obs data side of things. Once we have those then we can start some testing.

[delippi]

@SamuelDegelia-NOAA, yes I can generate whatever you need. A couple of questions.

1. Are these ctests only for MPAS-JEDI?
2. Just to make sure I understand, you just want one giant yaml with all the observations currently tests? That would be aircraft and mesonet at the moment.

[SamuelDegelia-NOAA]

@delippi Yep, I was thinking just starting with MPAS-JEDI and then creating one giant yaml with the current yamls (aircraft and mesonet). And then we can slowly expand it as we go forward.

[SamuelDegelia-NOAA]

GETKF and LETKF applications require the covariance localization to be set under the obs-space section of the yaml. So I will need to add that information in to each yaml in rrfs-test/validated_yamls/templates/obtype_config.

@delippi, @ShunLiu-NOAA, or @TingLei-NOAA would you be able to share where I can find the localization radii used for each obtype in RRFSv1? Just so that I can use realistic values for testing.

[guoqing-noaa]

I would provide an alternate view of the giant XML file.

We all understand that's the final XML file used in the DA task of rrfs-workflow. But I think we should cat them into one giant XML file at the last step in the workflow. Before that, we should tackle each obs type separately. That's much easier to handle and debug. NCAR has been running a small-domain cycling MPAS+JEDI experiment and they used a similar strategy. I got a chance to check their experiment run directories and I think it worked very well.

[SamuelDegelia-NOAA]

@guoqing-noaa The method you describe is very similar to what we are doing. Each obs type is handled individually in rrfs-test/validated_yamls/templates/obtype_config. We can adjust various settings for individual obs types in those split files.

For this issue specifically, we will cat these individual files together to build one mega-yaml for the ctest. A similar process will likely be used in the workflow but through the use of JCB instead of cat.

Also, right now I am just catting them together manually, but it is possible to add a command that runs during build.sh to create them.

[guoqing-noaa]

@SamuelDegelia-NOAA Thanks for the information. Is it possible to do the "cat" in the CTest step? That will be a cleaner solution.

Also, let's keep things simple and straightforward, just use cat until we have a formal decision on whether to adopt JCB in RDASApp or rrfs-workflow. I have lots of questions on JCB at the moment.

[SamuelDegelia-NOAA]

@guoqing-noaa I am not sure if it is possible to run cat inside the ctest. The ecbuild_add_test function seems to only be designed for running a single executable. We could do run the cat during the cmake step at the beginning of build.sh, but that is not much different from just including the cat inside build.sh directly.

[guoqing-noaa]

@guoqing-noaa I am not sure if it is possible to run cat inside the ctest. The ecbuild_add_test command seems to only be designed for running a single executable. We could do run the cat during the cmake step at the beginning of build.sh, but that is not much different from just including the cat inside build.sh directly.

Okay, then it is fine to cat inside build.sh as no better choices. Thanks!

[TingLei-NOAA]

GETKF and LETKF applications require the covariance localization to be set under the obs-space section of the yaml. So I will need to add that information in to each yaml in rrfs-test/validated_yamls/templates/obtype_config.

@delippi, @ShunLiu-NOAA, or @TingLei-NOAA would you be able to share where I can find the localization radii used for each obtype in RRFSv1? Just so that I can use realistic values for testing.

I think @ShunLiu-NOAA is at a better position to clarify on what cutoff radius are used in the current rrfs runs. From https://github.com/NOAA-EMC/rrfs-workflow/blob/main/parm/config/enkf/var_defns.sh#L784, when enkf of conventional data, for all obs (including ps and sat ), CORRLENGTH="300"
LNSIGCUTOFF="0.5" When EnkF using dbz, CORRLENGTH_radardbz="18" ,LNSIGCUTOFF_radardbz="0.5" . Namely, in current rrfs enkf, only dbz and non-dbz obs are treated differently.

[SamuelDegelia-NOAA]

@TingLei-NOAA Thank you for sharing! That should be sufficient for at least setting up some of our initial tests.