test the dual resolution 3DEnVAR capability

[guoqing-noaa]

See the rrfs-workflow ticket:

https://github.com/rrfs2/rrfs-workflow/issues/75

[TingLei-NOAA]

@guoqing-noaa Is it for mpas-jedi? For fv3-jedi, we also had an example hera :/scratch2/NCEPDEV/fv3-cam/Ting.Lei/dr-emc-regional_jedi/dr-rrfs_rundir/anla_conv_gsi_use_gsidiag_obs_newjedi/d-dualres-ens3dvar.sh. That will be great if there would also be a dual res test for mpas-jedi.

[guoqing-noaa]

Yes, we are working on MPASJEDI.

Thanks for your information on fv3jedi. We will check that.

[guoqing-noaa]

Results from @Junjun-NOAA . The dual resolution ensemble BEC works but the increments are not smooth when using lower-resolution ensembles. @TingLei-NOAA @SamuelDegelia-NOAA Do you have any suggestions to make the increments smooth in the dual-resolution situation? Thanks!

[TingLei-NOAA]

@guoqing-noaa @Junjun-NOAA Thanks for sharing. Great to see dual res works for regional mpas-jedi. To have a smoother increment from dual res, one thing it to use a larger "resolution" number in creating the bump correlation files on 24 km. That could create a sampling matching 12 km analysis resolutions. But this might reduce the benefits using dual res. Maybe we could dig more to see how the change res is done in the dual res run.

[guoqing-noaa]

@TingLei-NOAA For dual-res 3DEnVAR, what bump files we should use? Generated on the 12km or 24km ?

[TingLei-NOAA]

@guoqing-noaa i think there was a set of bump files generated for the localization on 24km domain as for 12km in the single resolution run.

[guoqing-noaa]

@TingLei-NOAA I think when we do 3DEnVAR, we can only use one set of bump files. Since the bump localization files are closed related to the ensembles used. I would assume in dual-res situation, we may need to use the bump files generated based on 24km members.

@Junjun-NOAA Could you do a test by re-generating 24km bump files and see whether this solves the unsmooth increment issue?

[TingLei-NOAA]

@guoqing-noaa Yes, in the dual res run, it is the bump files on 24km domain that is used for the ensemble localization.

[Junjun-NOAA]

@TingLei-NOAA @guoqing-noaa The results are from the 24km dump files. I tried to use the 12km bump files at first but the program did not pass.

[TingLei-NOAA]

@Junjun-NOAA Yes. as we discussed in the above, it should be 24km bump files to be used in your dual ens run. So , you might try increasing the "resolution" in your yaml to create that 24 km bump files to see if it help. Thanks.

[guoqing-noaa]

@TingLei-NOAA @guoqing-noaa The results are from the 24km dump files. I tried to use the 12km bump files at first but the program did not pass.

Sorry, @Junjun-NOAA. I did not know this. Since 24km bump files were used, we will check other solutions.

[Junjun-NOAA]

@TingLei-NOAA @guoqing-noaa I have tried different "resolution" values in yaml to create the 24 km bump files. The resulting increments are shown below: Any comments, please! Thanks

[TingLei-NOAA]

@Junjun-NOAA Thanks for sharing. That is great that apparently all are working under control:). Just one question, did you check the "Effective horizontal resolution " that would be printed out in the standard output from the bump app?
For the concern on the smoothness of the dual res results, I am also afraid that the change_resol (and its adjoint) methods implemented in mpas-jedi could also contribute to it ( change_resol_fields in "mpas_fields_mod.F90) . Maybe it is an outcome from such change_res on unstructured grids.

[Junjun-NOAA]

@TingLei-NOAA Thanks for your comments. I checked the "Effective horizontal resolution " in log file, here is the finding: when "resolution" is less than 8, the effective horizontal resolution is equal to "resolution"; when "resolution" is 8 and above, the effective horizontal resolution is 7.45. I will take a look at the change_resol (and its adjoint) methods and see what we can find. Thanks

[TingLei-NOAA]

@Junjun-NOAA I think we could also try to reach a larger effective resolution (like 10) by setting a larger "max horizontal grid size) (larger than default) as error_covariance_training_tutorial_2.yaml in saber's testinput. Also, an interesting relevant development is that JEDI core teams just have had A PR to update/improve change_resol for fv3-jedi https://github.com/JCSDA-internal/fv3-jedi/pull/1243#issuecomment-2270330066. It could also be considered for mpas-jedi in the future.

[fmahebert]

@TingLei-NOAA Thanks for bringing this thread to my attention. In addition to the PR in fv3-jedi, I have already opened one in mpas-jedi as well, see here: https://github.com/JCSDA-internal/mpas-jedi/pull/1002

I have not yet tested the mpas-jedi PR as carefully as the fv3-jedi PR. If you or @Junjun-NOAA are able to run experiments like the ones above using my PR, that could be a really helpful verification of the new interpolation algorithm... let me know what you think!

[TingLei-NOAA]

@fmahebert Thanks for this update. That is great you already have a PR on this for mpas-jedi. And I think @Junjun-NOAA or myself would be very happy to test it too using the case @Junjun-NOAA is working on.

[Junjun-NOAA]

@fmahebert Thanks for your update. We will have a test. @TingLei-NOAA, it seems I don't have access to JCSDA-internal, can you share the information there?

[TingLei-NOAA]

@Junjun-NOAA Sure. I will download that branch onto hera when it come back tomorrow and share with you all the details.

[TingLei-NOAA]

@Junjun-NOAA Sorry. on hera. I am having some issue building mpas-jedi from the current jedi-bundle and will let you know I finish it. The jedi-bundle with this PR is hera://scratch2/NCEPDEV/fv3-cam/Ting.Lei/dr-jedi-bundle-mpas-jedi-pr/jedi-bundle.

[Junjun-NOAA]

@TingLei-NOAA Thanks for the update.

[TingLei-NOAA]

@Junjun-NOAA I had just built a jedi-bundle with the changes in mpas-jedi. The executables are in /scratch2/NCEPDEV/fv3-cam/Ting.Lei/dr-jedi-bundle-mpas-jedi-pr/jedi-bundle/build/bin. I built it using gdasapp's modules (mainly). I think rdasapp is using the same modules. But extra caution is maybe needed. Please let me know if any problems.

[Junjun-NOAA]

@TingLei-NOAA Great. I will give it a try and keep you posted. Thanks a lot.

[Junjun-NOAA]

@TingLei-NOAA I copied the executable to Jet and loaded the same modules from RDASApp. The following message is given " error while loading shared libraries: libodccore.so: cannot open shared object file: No such file or directory". Do you know about this library? What module should I load? Thanks

[guoqing-noaa]

I think it is because Ting used the GDASApp module files. @TingLei-NOAA If it is not possible to use RDASApp module files to build this exec, could you show @Junjun-NOAA how to load your GDASApp module files? Thanks!

[TingLei-NOAA]

@Junjun-NOAA can you run them on hera? because in addition to gdasapp, I still use the newer fms lib on hera. I am not sure if they are available on jet. IF it is too time-consuming , would you please point me to your rundir and I could copy them to hera . @guoqing-noaa using gdasapp module is simple, just go to their modue dir and run module load as in rdasapp . However, I don't think gdasapp has support for jet. In fact, I think your added rdasapp module on jet could be used. But as I mentioned, a new fms is still needed. That is good to make it finally work on jet. But I would consider to run them on hera to have a quick test for the PR in question and see if it would improve the dual res results.
Please let me know how you want to proceed.

[Junjun-NOAA]

@TingLei-NOAA Thanks. I will copy my data to Hera and give it a try.

[Junjun-NOAA]

Using this PR, the increment is much smoother. Please see the figure below:

[TingLei-NOAA]

@Junjun-NOAA Great! Did you see any differences in the time for those related steps ?
Sorry you are still not able to check the jedi internal repo webpages. the below is the report from BJ on his global mpas case : Did you see similar behavior ?

[Junjun-NOAA]

@TingLei-NOAA Yes, there is some difference in the time for related steps, but not as big at that for BJ's case. Please see the below screen shot. For detailed information, you can see logs files located at: /scratch1/BMC/wrfruc/jjhu/rundir/RDASApp/expr/mpas_2024052700_12km24km/logs (Hera)

[guoqing-noaa]

Great! Thanks, @Junjun-NOAA and @TingLei-NOAA !

Junjun, do you plan to test it further on the 3km/12km dual res?

[TingLei-NOAA]

@Junjun-NOAA That is great! I will also attach the link to that mpas-jedi PR

[Junjun-NOAA]

@guoqing-noaa Yes, I will test 3km/12km also.

[TingLei-NOAA]

@guoqing-noaa @Junjun-NOAA Would you please point me to your 3km test case. I plan to use such a more realistic case to test/evaluate the saber/mgbf capabilities . Thanks.

[Junjun-NOAA]

Hi Ting, I had problem generating a CONUS 3km domain, the error is related to PIO, "ERROR: MPAS IO Error: PIO error -221: Overflow when type cast to 4-byte integer." I am assuming it is caused by the larger number of cells. So I am using a much smaller domain to test the dual-res capability. The increment can be seen from the following figure: I found that the increment is much small compared to that from 12km and 12km/24km experiments. I am looking at it. The spread looks good to me. I wonder if we should generate bump files using different setting other than that from 12km.

The run directory is /scratch1/BMC/wrfruc/jjhu/rundir/RDASApp/expr/mpas_2024052700_3km12km, subdirectory "data" has the background, ensemble, bump and obs data. "3km" are those fro 3km domain. Please let me know if you have any comments or questions. Thanks

[guoqing-noaa]

@Junjun-NOAA Thanks for the results! It is like a miracle that you can still use the 12km bumploc files for 3km DA. Could you try to generate 3km bumploc files? Follow section 6 of this wiki: https://github.com/NOAA-EMC/RDASApp/wiki/A-Quick-Start-Guide-for-Running-the-MPASJEDI-2024052700-case-on-Jet-or-Hera

Let me know if have any questions.

[TingLei-NOAA]

@Junjun-NOAA Thanks a lot for sharing ! And that is great you resolved those PIO error by using more mpi numbers. So, for 3km analysis, you are using 12 km ensembles, right? That will be interesting if still 24 km ensembles are used and see if we could see similar results (maybe the observation need to be adjusted to have a similar omb first).

[Junjun-NOAA]

@guoqing-noaa I regenerated the bump files. I only changed the geometry and background setting. I meant I did not change anything in background error section. I am not sure if we should change different settings.

[guoqing-noaa]

@Junjun-NOAA Thanks for the clarification. The 3km background changed, maybe the max increment level changed as well. Did you check different vertical levels? If you ncdump the hofx.nc4 file, what is the OMA?

[Junjun-NOAA]

@guoqing-noaa I checked several levels from 15 to 30, let me check the max and min values in each level. The OMA and OMB values are:

The group: oman { variables: float airTemperature(Location) ; airTemperature:_FillValue = -3.368795e+38f ; data:

airTemperature = 5.819345 ; } // group oman

group: ombg { variables: float airTemperature(Location) ; airTemperature:_FillValue = -3.368795e+38f ; data:

airTemperature = 6.057054 ; } // group ombg

[Junjun-NOAA]

@TingLei-NOAA Yes, I did test using 3km ensemble and 12km ensemble separately.

[Junjun-NOAA]

@guoqing-noaa @TingLei-NOAA I have checked the max and min temperature increment values in each vertical level. Below I list the biggest increment for 3km and 12km single resolution experiments: 3km: level = 22, max = 0.33 ,min = -0.01 12km: level = 18, max = 2.15 ,min = -0.12

The OMB and OMA values for them are: 3km: OMB = 6.06, OMA = 5.82 12km: OMB = 5.90, OMA = 4.53

The OMB values are quite similar, increments have much difference.

[TingLei-NOAA]

@Junjun-NOAA Thanks for sharing! The first look at those results, the different locations of the maximum increments by 4 levels is not easy to understand. Did you see the vertical levels for dual 3km/12km (and 12km/24km)? if the original differences came from vertical covariances in 3km vs 12 km ensembles. we should see 3km/12km (ens) have similar locations as in 12km single res .

[Junjun-NOAA]

@TingLei-NOAA Thanks for your comments. I checked the increments from both 3km single-res and 3km/12km dual-res experiments. The biggest increment from both experiment are at the same vertical level (lev=22, index from 0 to 54). I think it is fine. I just don't understand why the 3km run is much smaller than 12km run.

[Junjun-NOAA]

Now I have successfully created a big COUNS 3km domain on Hera. The ensemble spread is much similar as that from 12km runs, see the following figure.
. However, the increment is much smaller than the 12 km runs. I quite not understand. . I have the small domain run on Hera: /scratch1/BMC/wrfruc/jjhu/rundir/RDASApp/expr/mpas_2024052700_3km12km_smalldomain and bigger domain at: /scratch1/BMC/wrfruc/jjhu/rundir/RDASApp/expr/mpas_2024052700_3km12km If someone has any comments, please let me know. Thanks

[TingLei-NOAA]

@Junjun-NOAA Thanks. So, did 12km/12km also have the maximum center on 22th level (it should)? Could we also see the oma ?

[Junjun-NOAA]

@TingLei-NOAA 12km/12km and 12km/24km have he maximum center on 18th level. 12km/12km: OMB = 5.90, OMA = 4.53; 12km/24km: OMB = 5.90, OMA = 4.57

[TingLei-NOAA]

@Junjun-NOAA Thanks for the clarification. So, that is the one more confusing/concerning to me: using the same sampling ensemble (and the same localization ), 12km/12km have different maximum center levels vs 3km/12km. I will see if I could take closer look next week. Thanks.

[Junjun-NOAA]

@TingLei-NOAA After creating an new obs by placing it near the maximum spread, the results look more reasonable now. Please take a look at the plots below. Thanks @guoqing-noaa for help creating the new obs.