NOAA-EMC / RDASApp

Regional DAS
GNU Lesser General Public License v2.1
0 stars 5 forks source link

Added experiment templates for running fv3jedi and mpasjedi on Orion. #29

Closed SamuelDegelia-NOAA closed 2 weeks ago

SamuelDegelia-NOAA commented 2 weeks ago

The goal of this PR is to add experiment templates for running fv3jedi and mpasjedi cases on Orion, and to unify the tests between fv3jedi and mpasjedi with both now featuring the mesonet single observation test.

Finally, following Donnie's example in his last PR, I created my own fork of the main repository to edit the Wiki to support the above changes: Wiki link. Orion is no longer listed as "coming soon", and there are some slight edits to the last section to indicate the new plotting tools for mpasjedi.

SamuelDegelia-NOAA commented 2 weeks ago

Thanks for reviewing @delippi. I also noticed the larger increments when testing things. The o-b is larger in the MPAS case but that alone likely doesn't explain the order of magnitude difference in the increments. Since the obs. errors are also similar, there is likely a large difference in the ensemble spread between MPAS and FV3. I'll have a look at this before merging so that we understand the difference a little more.

delippi commented 2 weeks ago

@TingLei-NOAA, can you think of any differences between the FV3 and MPAS experiments that might be leading to such a large difference between them with the single ob test? Were backgrounds overwritten as analyses at some point or anything of that nature? How hard is it to regenerate a clean MPAS experiment to check? I would think the only major difference should be which dycore was used and that it should produce a similar increment, hofx, etc. It would be good to make sure both cases can produce [more] similar results which will be helpful for future sanity checks.

SamuelDegelia-NOAA commented 2 weeks ago

One main difference here is that the fv3jedi case is hybrid with 10% weight to static covariances, and 90% weight to ensemble covariances computed with 20 ensemble members. The mpasjedi case is fully EnVar without any static covariances, and only features 5 ensemble members. So these tests are configured pretty differently.

SamuelDegelia-NOAA commented 2 weeks ago

A large difference in spread, along with the larger o-b values in mpasjedi, likely explain the large difference in increments. Below is a comparison of temperature spread (standard deviation) at the lowest model level between fv3jedi (left) and mpasjedi (right). The spread in mpasjedi is ~3x larger than fv3jedi which would weight the single observation much more.

This result is thus likely behaving as expected considering the difference in experiment designs. If we had 20 MPAS members, the spread might be lower and the increments between the two experiments might be closer together. Whether we want to bring the mpasjedi configuration closer to fv3jedi, I will leave that up to the rest of the group.

Screen Shot 2024-04-30 at 4 11 44 PM
delippi commented 2 weeks ago

Thanks @SamuelDegelia-NOAA. I think this PR is good to go. We should discuss with @TingLei-NOAA on how to bring the MPAS configuration closer to the FV3, but that is out of the scope of this PR.

ShunLiu-NOAA commented 2 weeks ago

Agree with Sam. The difference are expected. We may merge the PR first. Once we can begin MPAS retro run, we will concentrate on creating B similar to that in GSI and FV3LAM. Then we can further compare the results.

TingLei-NOAA commented 2 weeks ago

@TingLei-NOAA, can you think of any differences between the FV3 and MPAS experiments that might be leading to such a large difference between them with the single ob test? Were backgrounds overwritten as analyses at some point or anything of that nature? How hard is it to regenerate a clean MPAS experiment to check? I would think the only major difference should be which dycore was used and that it should produce a similar increment, hofx, etc. It would be good to make sure both cases can produce [more] similar results which will be helpful for future sanity checks.

@delippi Sorry. just saw your message. I think they are good explanation mentioned in those following discussion. It is a little difficult for us to get a more similar result between the mpas-jedi and fv3-jedi. Because in the former, now the ensemble of size 5 are got from 5 forecast on different time levels and we don't have conversion tools , at least for being now, to do conversion between mpas and fv3. I think this is a question/issue (no more "exact" comparison) we need bear with for some time.