Adding dust from GEFS-Aerosol to use as LBC for RRFS

[perthsb]

Adding the capability to use dust from the global aerosol model at EMC (GEFS-Aerosol) use it as LBC for RRFS (Smoke and dust). Similar setup done for CMAQ as well . Work needed to update the workflow (ARL is going to help on this) followed by testing the model.

[JacobCarley-NOAA]

@chan-hoo is this something you might be able to assist with? Thanks!

[chan-hoo]

@perthsb, you described above ARL would work on this. Is this correct? If you need any assistance from me, please let me know.

[perthsb]

Thanks Chan-Hoo. Adding @bbakernoaa at ARL in this to add any technical details.

[bbakernoaa]

@chan-hoo specifically can you help with getting the workflow jobs and such ported over from AQM to the RRFS workflow. It would be very helpful to have that. Once done it is easy for us to go in an adjust the details.

[bbakernoaa]

@ytangnoaa

[chan-hoo]

@bbakernoaa, @ytangnoaa, which job script in the AQM (aqm_dev) workflow (https://github.com/ufs-community/ufs-srweather-app/tree/aqm_dev/jobs) is related to this?

[bbakernoaa]

@chan-hoo Specifically it is https://github.com/ufs-community/ufs-srweather-app/blob/aqm_dev/jobs/JAQM_AQM_LBCS

and even more specifically its just the add GEFS-lbcs parts such as https://github.com/ufs-community/ufs-srweather-app/blob/aqm_dev/scripts/exaqm_aqm_lbcs.sh#L236-L264

[ytangnoaa]

@bbakernoaa, @ytangnoaa, which job script in the AQM (aqm_dev) workflow (https://github.com/ufs-community/ufs-srweather-app/tree/aqm_dev/jobs) is related to this?

It should be related to https://github.com/ufs-community/ufs-srweather-app/blob/aqm_dev/jobs/JAQM_AQM_LBCS However, RRFS uses much thicker halo layers (Halo=24), which has no corresponding grid file (we only have halo4 et al).
We can not use current UFS-AQM script/code directly

[chan-hoo]

@ytangnoaa, do you mean that you need a 24-halo grid file for this task?

[ytangnoaa]

@ytangnoaa, do you mean that you need a 24-halo grid file for this task? Yes, if it can be generated.

[chan-hoo]

@ytangnoaa, I think we can. I have not tried it before though. Which domain do you use for this task? RRFS_CONUS_3km? RRFS_NA_3km?

[bbakernoaa]

@chan-hoo it would be good if both could be created. The ultimate target would be RRFS_NA_3km

[chan-hoo]

Got it. I'll try it soon (hopefully :) )

[chan-hoo]

@ytangnoaa, can you test the following grid?

/scratch2/NCEPDEV/fv3-cam/Chan-hoo.Jeon/rrfs_halo24/expt_dirs/test_nonDA_conus3km/grid/C3359_grid.tile7.halo24.nc

Please let me know if it is what you need.

[ytangnoaa]

@ytangnoaa, can you test the following grid?

/scratch2/NCEPDEV/fv3-cam/Chan-hoo.Jeon/rrfs_halo24/expt_dirs/test_nonDA_conus3km/grid/C3359_grid.tile7.halo24.nc

Please let me know if it is what you need.

Chan-hoo. Thank you for your file. Do you have the corresponding "C3359_oro_data.tile7.halo24.nc" file with the lat/lon and terrain data? I also need the input LBC file with the meteorological variables. The dust/fire tracers will add upon that LBC file. @bbakernoaa Could you help get a sample file for testing?

[bbakernoaa]

@jordanschnell @JacobCarley-NOAA Can you help with providing an example of an input file where the tracers are included for the large NA domain?

[chan-hoo]

@ytangnoaa, @bbakernoaa I found a critical issue on the grid with 24 halo. By the nature of the ESG grid generator, the optimal values of two parameters (A,K) are changed by the number of cells. This means that the main inner areas (without halos) of two grids with halo6 and halo24 are not identical. They are slightly different. For example, in RRFS_CONUS_25km, halo6: A=0.1255, K=-0.3366, but halo24: A=0.1524, K=-0.3076. Accordingly, Cres,halo6=403, but Cres,halo24=405. If you need an identical grid for the inner area, we'll need a help from the developer of the ESG grid tool. Please let me know what you think.

[ytangnoaa]

@ytangnoaa, @bbakernoaa I found a critical issue on the grid with 24 halo. By the nature of the ESG grid generator, the optimal values of two parameters (A,K) are changed by the number of cells. This means that the main inner areas (without halos) of two grids with halo6 and halo24 are not identical. They are slightly different. For example, in RRFS_CONUS_25km, halo6: A=0.1255, K=-0.3366, but halo24: A=0.1524, K=-0.3076. Accordingly, Cres,halo6=403, but Cres,halo24=405. If you need an identical grid for the inner area, we'll need a help from the developer of the ESG grid tool. Please let me know what you think.

Chan-Hoo, thank you for your effort. For the LBC interpolation purpose, we need the LBC halo layers' 3-D location: lat/lon/elevation. How big difference of the inner domains between halo6 and halo24?

[chan-hoo]

@ytangnoaa, you can find some differences on RRFS_CONUS_25km here:

/scratch2/NCEPDEV/fv3-cam/Chan-hoo.Jeon/tools/fv3sar_pre_plot/Fig

[ytangnoaa]

@ytangnoaa, you can find some differences on RRFS_CONUS_25km here:

/scratch2/NCEPDEV/fv3-cam/Chan-hoo.Jeon/tools/fv3sar_pre_plot/Fig

Thank you, Chan-hoo. The location shift error could be more than one grid cell. Does RRFS have to use this thick halo boundary? UFS-AQM online CMAQ used a thicker boundary, but later changed to halo=4 to fit with the existing grid files

[chan-hoo]

@ytangnoaa, I expect that the 3km domain would have much smaller difference. I'll check them out and let you know.

[chan-hoo]

@ytangnoaa, you can find the difference plot for RRFS_CONUS_3km here:

/scratch2/NCEPDEV/fv3-cam/Chan-hoo.Jeon/tools/fv3sar_pre_plot/Fig/fv3lam_comp_RRFS_CONUS_3km_h001_geolon.png

Its difference is about one cell size near the edges. By the way, I heard that there might be a tool (function) for LBC in GSI so that we would not need 24 halos. I am not sure if this is correct. Can you point me to the part (in a script) where RRFS uses 24 halos?

[ytangnoaa]

@ytangnoaa, you can find the difference plot for RRFS_CONUS_3km here:

/scratch2/NCEPDEV/fv3-cam/Chan-hoo.Jeon/tools/fv3sar_pre_plot/Fig/fv3lam_comp_RRFS_CONUS_3km_h001_geolon.png

Its difference is about one cell size near the edges. By the way, I heard that there might be a tool (function) for LBC in GSI so that we would not need 24 halos. I am not sure if this is correct. Can you point me to the part (in a script) where RRFS uses 24 halos?

Chan-hoo. You can find my sample script for extracting GEFS-Aerosol nemsio output to supply RRFS LBC in

/scratch2/NCEPDEV/fv3-cam/noscrub/Youhua.Tang/aqm_parallel_glbc.fd/rrfs-sd/test-C3463.ksh

It can use the existing AQM_util's executable file, with different output variables: 'dust' and 'coarsepm'. The halo=24 information comes from the inputted meteorological LBC files, and the dust LBC is supposed to be added upon them. We need supply the lat/lon/elevation information to extract GEFS-Aerosol output, but the existing file

topofile='C3463_oro_data.tile7.halo4.nc'

can only support halo4 grid

[chan-hoo]

@ytangnoaa, I think there was a confusion between halo_blend and halo_bc. In RRFS, halo_blend=20 and halo_bc=4. However, this doesn't mean that total halo number outside the main domain is 24. halo_blend means the number of cells "inside" the main domain for blending. So you can use the "halo4" grid.

[ytangnoaa]

@ytangnoaa, I think there was a confusion between halo_blend and halo_bc. In RRFS, halo_blend=20 and halo_bc=4. However, this doesn't mean that total halo number outside the main domain is 24. halo_blend means the number of cells "inside" the main domain for blending. So you can use the "halo4" grid.

Chan-hoo, In the meteorological LBC files, there is only dimension of 'halo=24', and the array is defined like

float zh_bottom(levp, halo, lon)

We still need to fill the array with halo=24 anyway. If the outside 20 layers are not used, why do not we simply only keep the inside 4 layers?

[chan-hoo]

Yes, the LBC files include both halo_blend (inside) and halo_bc (outside). What I mean is that you can get the grid info like lon/lat for the 24 halos from the halo4 grid file. The 24 cells from the outer boundary in the halo4 grid are the halo points.

[chan-hoo]

@ytangnoaa, I hope the below figure (plot of gfs_bndy.tile7.000.nc in RRFS) will help you better understand the concept of blending and halo. In the figure, the white dashed lines are the boundary of the main domain (halo0). As you can see, the LBC files include both halo and blending areas. This is the reason why its dimension is 24 (halo=4, blending=20).

[ytangnoaa]

Chan-hoo. Thank you for your information. So, the RRFS's halo includes the inside blended layers, and this information needs to be provided to the LBC process. I included it in the namelist file (inblend=20). Please look at

/scratch2/NCEPDEV/fv3-cam/noscrub/Youhua.Tang/aqm_parallel_glbc.fd/rrfs-sd gefs2lbc_para.f90 Makefile test-C3463.ksh

I have done the engineering test, and the dust LBC output (dust and coarsepm in unit ug/kg) looked reasonable

/scratch2/NCEPDEV/fv3-cam/noscrub/Youhua.Tang/aqm_parallel_glbc.fd/rrfs-sd/tmp

Could someone help to verify its impact? @bbakernoaa

[chan-hoo]

@ytangnoaa, Is gefs2lbc_para the same as that in AQM-utils?

[ytangnoaa]

@ytangnoaa, Is gefs2lbc_para the same as that in AQM-utils?

It has some differences from AQM-utils, such as the namelist variable "inblend" to tell the blended layers inside the domain. We'd better change it to another filename

[chan-hoo]

Got it. If it is compatible with the AQM workflow, I think it would be better to upgrade gefs2lbc_para in AQM-utils for use in both AQM and RRFS because the AQM workflow will be incorporated into the RRFS workflow in the near future. What do you think about this? Do you plan to add it to rrfs_utl rather than AQM_utils?

[ytangnoaa]

Got it. If it is compatible with the AQM workflow, I think it would be better to upgrade gefs2lbc_para in AQM-utils for use in both AQM and RRFS because the AQM workflow will be incorporated into the RRFS workflow in the near future. What do you think about this? Do you plan to add it to rrfs_utl rather than AQM_utils?

I updated the code in AQM_utils. So they can share the same code.

https://github.com/noaa-oar-arl/AQM-utils/commit/2c4415516b01abd8c7b5e6c1e663798099578cbe

Its default inblend=0 (in current UFS-AQM's C793 domain). For RRFS-NA3km, the namelist is added with inblend=20

[chan-hoo]

@ytangnoaa, can you update the authoritative repository of AQM-utils (NOAA-EMC/AQM-utils)?

[ytangnoaa]

@ytangnoaa, can you update the authoritative repository of AQM-utils (NOAA-EMC/AQM-utils)?

I have no permission to change NOAA-EMC/AQM-utils. I updated the PR and forked repository https://github.com/NOAA-EMC/AQM-utils/pull/12#issuecomment-1860153643

[ytangnoaa]

Here is the testing result for one run: 2023022600, https://docs.google.com/presentation/d/111aWLKqJ6uKc3-DWwigtF3OesogiKAaB/edit?usp=sharing&ouid=115981223134185527389&rtpof=true&sd=true @JacobCarley-NOAA @bbakernoaa @chan-hoo

[perthsb]

Thanks @ytangnoaa . @JacobCarley-NOAA anything you want to add before @chan-hoo proceeds further on this.

[JacobCarley-NOAA]

Thanks @ytangnoaa . @JacobCarley-NOAA anything you want to add before @chan-hoo proceeds further on this.

Nothing from my end. Thanks @ytangnoaa for the slides and testing.

[perthsb]

@ytangnoaa , quick question to you on the factor you have used for unit conversion from GEFS to RRFS shows in the slide. Based on that it seems to be Bin 2 is multiplied by 2 different factors and split into coarse and fine dust pm in RRFS. Is that same done with CMAQ as well ?

[ytangnoaa]

@ytangnoaa , quick question to you on the factor you have used for unit conversion from GEFS to RRFS shows in the slide. Based on that it seems to be Bin 2 is multiplied by 2 different factors and split into coarse and fine dust pm in RRFS. Is that same done with CMAQ as well ?

Yes, the aerosol size mapping from GEFS to CMAQ uses the similar approach, though with different splitting factors

[perthsb]

@chan-hoo @JacobCarley-NOAA Any update on this issue, whether workflow has been updated accordingly and when it can go in realtime RRFS_A. GSL is making smoke PR and would like to know what changes need to be added for dust in this table : sorc/UFS_UTILS/parm/varmap_tables/GSDphys_var_map.txt

[chan-hoo]

@perthsb, I had to fix the issues on the sample eng test script and am setting up a new ensemble eng test script for future development. Once I complete this, I'll get back to this issue soon.

[perthsb]

Thanks @chan-hoo .

[perthsb]

@ytangnoaa I am wondering whether you can provide any help on the table values in sorc/UFS_UTILS/parm/varmap_tables/GSDphys_var_map.txt with the change for dust using LBCs from GEFS-Aero. Currently it is set as smoke smoke set_to_fill 1E-12 T coarsepm coarsepm set_to_fill 1E-12 T dust dust set_to_fill 1E-12 T

[ytangnoaa]

@ytangnoaa I am wondering whether you can provide any help on the table values in sorc/UFS_UTILS/parm/varmap_tables/GSDphys_var_map.txt with the change for dust using LBCs from GEFS-Aero. Currently it is set as smoke smoke set_to_fill 1E-12 T coarsepm coarsepm set_to_fill 1E-12 T dust dust set_to_fill 1E-12 T

Just to clarify. The dust species mapping from the GEFS-Aerosol follows the method of

https://github.com/ufs-community/ccpp-physics/blob/ufs/dev/physics/smoke_dust/dust_fengsha_mod.F90#L364-L365

It won't affect the default fill values. If the GEFS-Aerosol LBC is not available (variables of smoke/dust/coarsepm are not appeared in the LBC file), RRFS-SD can automatically use these fill values.

[ytangnoaa]

@perthsb, I had to fix the issues on the sample eng test script and am setting up a new ensemble eng test script for future development. Once I complete this, I'll get back to this issue soon.

@chan-hoo Another issue about the workflow. The meteorological GFS LBC are in hourly files, e.g. /scratch2/NCEPDEV/stmp3/Wei-ting.Hung/RRFS_dust/T.CONUS.3km.20240108/nwges/2024010800/lbcs However, the GEFS-aerosol has output every 3 hours. Are we required to use the hourly LBC?

[chan-hoo]

@ytangnoaa, I don't have an answer to this. @JacobCarley-NOAA @MatthewPyle-NOAA, any thought?

[bbakernoaa]

@chan-hoo I think that this is just the limitation from the GEFS output frequency. Over a 3 hour period the overall large scale transport isn't going to change much. We could use persistence for the 3 hours or do a time interpolation and be fine. I think @ytangnoaa is already working on it.

[perthsb]

@bbakernoaa how is that handled in CMAQ which also uses GEFS-Aerosol

[ytangnoaa]

@bbakernoaa how is that handled in CMAQ which also uses GEFS-Aerosol

We are using 3-hours GEFS-Aerosol LBC for both offline CMAQ and online UFS-AQM

[chan-hoo]

I am back to this issue :)