Modification of the linoz v3 chemistry interpolation method to better conserve chemical mass

[jinboxie]

The current PR documents the modification of the chemistry interpolation method to better conserve chemical mass. It is also used to test the new forcing linoz_v3 data produced. Please refer to confluence for the production of the linoz_v3 new forcing data. https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/3764486280/Production+of+the+Linoz+v3+data.

The current NGD_v3atm uses default interpolation method from e3sm that linear interpolation (fraction of upper and lower end based on inverse distance). We utilize the UCI interpolation box method and implement into e3sm to better conserve mass. Three groups of results using two year simulation (shown below) indicates that the interpolation does not make significant difference between the simulations. Due to linoz v3 old forcing does not have future scenario, we compared the future SSPs with the linoz_v2 in E3SM maint2.0 instead. The three groups of simulation are as follows: 1.v3_historical v3atm.historical_Linoz-v3_UCI: used with linv3 new forcing and ran with new UCI interpolation (1850-1851) v3atm.historical_Linoz-v3 : used with old linv3 forcing made by Juno Hsu and linear interpolation (1850-1851) 2.v2_SSP370 maint2.0.historical_Linoz-v2_SSP370: used with original e3smv2 CMIP6 SSP370 forcing and linoz v2 (2020-2021) master.historical_Linoz-v2_SSP370 : the same forcing with above and new linoz focing with UCI interpolation (2020-2021) 3.v2_SSP585 maint2.0.historical_Linoz-v2_SSP585: used with original e3smv2 CMIP6 SSP585 forcing and linoz v2 (2020-2021) master.historical_Linoz-v2_SSP585 : the same forcing with above and new linoz focing with UCI interpolation (2020-2021) For a explicit documentation, please refer to linoz v3 simulation documentation v3_historical v2_SSP370 v2_SSP585

[tangq]

@jinboxie , was the file linv3_uci72lvs_1849-2015_2010JPL_cmip6_historical_10deg_58km_c20230312.nc uploaded to the E3SM input data server?

[tangq]

@jinboxie , I looked through the code changes. A general comment is to remove all the tagged comments and print/write statements. Git saves all these details automatically, so E3SM don't encourage these.

@wlin7 , the changes in this PR will affect tests using F20TR, F1850, and F2010 compsets and other tests depending on the use case of these compsets. What test commands should @jinboxie run to confirm the changes are good to be merged? Thanks.

[jinboxie]

@tangq Hi Qi, the files are moved here /lcrc/group/acme/public_html/inputdata/atm/cam/chem/trop_mozart/ub/linv3_uci72lvs_1849-2015_2010JPL_cmip6_historical_10deg_58km_c20230312.nc

[tangq]

@tangq Hi Qi, the files are moved here /lcrc/group/acme/public_html/inputdata/atm/cam/chem/trop_mozart/ub/linv3_uci72lvs_1849-2015_2010JPL_cmip6_historical_10deg_58km_c20230312.nc

Can you make the file group writable by the e3sm group? Also, add a link to the instructions on how to create this input file here.

[jinboxie]

Hi Qi @tangq . This data is initially moved to /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mozart/ub/linv3_uci72lvs_1849-2015_2010JPL_cmip6_historical_10deg_58km_c20230312.nc and somehow picked by ac.ngmahfouz to move to /lcrc/group/acme/public_html/inputdata/atm/cam/chem/trop_mozart/ub/linv3_uci72lvs_1849-2015_2010JPL_cmip6_historical_10deg_58km_c20230312.nc I dont seem to have the permission to modify it.

[tangq]

@jinboxie , can you change the permission for the origional copy at /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mozart/ub/linv3_uci72lvs_1849-2015_2010JPL_cmip6_historical_10deg_58km_c20230312.nc? If not, you may reach out to the LCRC admin to help with this.

[jinboxie]

@jinboxie , can you change the permission for the origional copy at /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mozart/ub/linv3_uci72lvs_1849-2015_2010JPL_cmip6_historical_10deg_58km_c20230312.nc? If not, you may reach out to the LCRC admin to help with this.

Hi Qi, it is changed.

[jinboxie]

@jinboxie , I looked through the code changes. A general comment is to remove all the tagged comments and print/write statements. Git saves all these details automatically, so E3SM don't encourage these.

@wlin7 , the changes in this PR will affect tests using F20TR, F1850, and F2010 compsets and other tests depending on the use case of these compsets. What test commands should @jinboxie run to confirm the changes are good to be merged? Thanks.

Hi Qi @tangq , the tags and statements are modified, and the new commit is produced and pushed to the branch. Hi Wuyin @wlin7, how should we test the new code using the related compsets?

[tangq]

@jinboxie , the remaining unresolved comments are about components/eam/src/physics/cam/tropopause.F90. I am not sure why your Linoz input changes affect tropopause.F90.

[jinboxie]

Hi Qi @tangq and Wuyin @wlin7 , the current branch is updated with original NGDv3 and added modifications with my change. It is two files with the UCI related interpolation algorithm. The test results shown in attached that there are no significant difference between the new and old algorithm.

[tangq]

Hi Qi @tangq and Wuyin @wlin7 , the current branch is updated with original NGDv3 and added modifications with my change. It is two files with the UCI related interpolation algorithm. The test results shown in attached that there are no significant difference between the new and old algorithm.

@jinboxie , the commit is clean now, but where is the attached figures you referred to?

[jinboxie]

Hi Qi @tangq and Wuyin @wlin7 , the current branch is updated with original NGDv3 and added modifications with my change. It is two files with the UCI related interpolation algorithm. The test results shown in attached that there are no significant difference between the new and old algorithm.

@jinboxie , the commit is clean now, but where is the attached figures you referred to?

Hi Qi @tangq , it was attached. I put the figures on the comments now so you can see without downloading.

[tangq]

@jinboxie , I see the figures now - You attached them to the first message instead of the last. It is okay.

The units are not labelled. Are the relative changes smaller than about 0.1%? or 10%?

[jinboxie]

Hi Qi, it is 10%. @tangq

[tangq]

Hi Qi, it is 10%. @tangq

@jinboxie , I don't remember the magnitudes of differences you showed when you first tested these changes. Are these results consistent with your previous tests based on the older code?

[jinboxie]

@tangq Hi Qi, I don't have the LNZ results for the previous version, attached is the previous O3 difference. As you can see from the image, it is consistent with our current testing results. The difference are in percentage.

[tangq]

@jinboxie , do you need to bypass the vertical interpolation in the old code? I am not sure which part of the changes does that.

Hi Qi

[jinboxie]

@jinboxie , do you need to bypass the vertical interpolation in the old code? I am not sure which part of the changes does that.

Hi Qi, please refer to linoz_data.F90, in the call for advance of trcdata (advance calculation for every timestep), I used a separate call advance_trcdata_linoz. This keeps original structure but uses the new interpolation method within advance_trcdata_linoz rather than the old one. A separate function advance_trcdata_linoz is also put in linoz_data.F90 Please refer to line 351 in old file and line 356,357 in new file of linoz_data.F90

[tangq]

@jinboxie , I posted the inline comment above. It seems exactly what you referred to. Can you reply to the inline comment above? It is not obvious what your changes are between advance_trcdata_linoz and advance_trcdata.

[jinboxie]

Hi Qi @tangq I found that this version has not any change within the code. It is updated with a new version now. It has the new UCI interpolation embedded within the interpolate_trcdata in tracer_data.F90. The file%linoz_v3 is also added to use the UCI algorithm for linoz_v3 variables for in interpolate_trcdata in tracer_data.F90. Minor changes are to advance_trcdata in linoz_data.F90 to set the linoz_v3 variable. The new code embed the UCI algorithm within the model, thus need not change to our old inputdata of 'linv3_1849-2015_2010JPL_cmip6_historical_10deg_58km_c20210625.nc'. The two years results figure are updated (top of the pull request). The figures are shown in the order of 1.new, 2.old, 3.new-old, 4.new-old (percentage, 0.05 noted as 5%). From the results, it can be seen that apart from the higher upper layer in CH4LNZ and N2OLNZ, the results are all minor.

[tangq]

@jinboxie , looks like you forgot the changes to the compset (xml) files.

[jinboxie]

@jinboxie , looks like you forgot the changes to the compset (xml) files.

Hi Qi, I input a newer version of the code with the interpolation embedded within the model. Now this code does not need any new input data since we do interpolation within the code. The tested results are also updated, which shows no difference from our previous work.

[tangq]

But we would like to use the new, clean Linoz input format, not to reuse the old one, which includes sloppy paddings.

[jinboxie]

Hi Qi @tangq , I have uploaded a new version of the current codebase. It has the new vertical interpolation embedded within code. For data surface boundaries, I have also added new variables and it takes over within the code. This surface takeover is done in model as the code has every CPU to readin the global data, and interpolate separately to local chunk lat/lon e3sm grid. We do the takeover process before interpolation so that we can get everything bit-for-bit. I have checked with the old simulations posted above and the results are the same. Next step, I'll check the model on linoz-v2 simulations.

[jinboxie]

@tangq Hi Qi, i've tested the code on linoz_v2, and it works fine. The model does what it does for the linoz_v3, and skips the variables not present in linv3 inputdata. The only issue is with the compilation of the code. I had to first make a compile with the original code of v3atm and replace with the new code. I'll check this final issue before completing the PR.

[jinboxie]

@tangq Hi Qi, the new code is modified and tested. It now works for both v3 and v2.

[jinboxie]

Besides these changes/clarifications. You may want to update the old vs. new results attached above for both Linoz v2 and v3.

@tangq Please correct me if I'm wrong. I'm trying to output O3, H2OLNZ, N2OLNZ, NOYLNZ, CH4LNZ. For the latter 4, which are the correspondent variables in linoz_v2?

[tangq]

For Linoz v2, the only prognostic variable is O3. You only need to show O3 results.

[tangq]

I don't think you need to force-push every time. You may do additional commits instead.

[tangq]

@jinboxie , once you updated the old vs. new figures and confirmed the results are reasonable. I will approve.

[jinboxie]

I don't think you need to force-push every time. You may do additional commits instead.

@jinboxie , once you updated the old vs. new figures and confirmed the results are reasonable. I will approve.

Ok, Qi. The simulations are hanged on chrysalis. We may wait a little time for the results.

[jinboxie]

@jinboxie , once you updated the old vs. new figures and confirmed the results are reasonable. I will approve.

Hi Qi. The data above is update for linoz_v2 O3. The results show that there are no significant difference between the 2 alogorithms.

[jinboxie]

Hi Qi, The suggested changes are implemented. Best, Jinbo

From: tangq @.> Date: Saturday, April 8, 2023 at 12:09 PM To: E3SM-Project/v3atm @.> Cc: Xie, Jinbo @.>, Mention @.> Subject: Re: [E3SM-Project/v3atm] Modification of the linoz v3 chemistry interpolation method to better conserve chemical mass (PR #62)

@tangq approved this pull request.

The results look reasonable. Approve when the suggested changes are implemented.

---

In components/eam/src/chemistry/mozart/lin_strat_chem.F90https://urldefense.us/v3/__https:/github.com/E3SM-Project/v3atm/pull/62*discussion_r1161146213__;Iw!!G2kpM7uM-TzIFchu!nr8IPA4Ln0ogiA0GVBehxoIbcPNit-aKKU2n5dU7-p0voFX-zhf4oc_im6Ozd3w$:

@@ -188,6 +188,17 @@ subroutine lin_strat_chem_inti(phys_state)

!

! initialize the linoz data

• !!set true for linoz_v3 interpolation in interpolation

• if (linoz_v3) file%linoz_v3=.true.

• if (linoz_v2) file%linoz_v2=.true.

• !!

• if (linoz_v2.eq..true. .and.linoz_v3.eq..true. ) then

• write(iulog,*) 'linoz_readnl, linoz: Both linoz_v2 and linoz_v3 are true. This is wrong! please check.'

• return

• elseif (linoz_v2.eq..false..and.linoz_v3.eq..false.) then

• write(iulog,*) 'linoz_readnl, linoz: Both linoz_v2 and linoz_v3 are false, please check if these setting are correct.'

Let's make this message clearer when both conditions are false. Change to something like "... Both linoz_v2 and linoz_v3 are false, which can be correct, but be sure that is intended!"

---

In components/eam/src/chemistry/utils/tracer_data.F90https://urldefense.us/v3/__https:/github.com/E3SM-Project/v3atm/pull/62*discussion_r1161146970__;Iw!!G2kpM7uM-TzIFchu!nr8IPA4Ln0ogiA0GVBehxoIbcPNit-aKKU2n5dU7-p0voFX-zhf4oc_iWBLfHio$:

• do i=1,ncol

• do k=1,pver

• call vert_interp_uci_single(pint(i,k+1),pint(i,k),dataout(i,k),pin,datain(i,:),levsiz)

• enddo

• enddo

Fortran uses column-major indexing, i.e., the fastest varying index is always the innermost. The innermost index should be the innermost loop in a nested loop for better performance. I recommend you swap the order of the two loops.

— Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https:/github.com/E3SM-Project/v3atm/pull/62*pullrequestreview-1376795932__;Iw!!G2kpM7uM-TzIFchu!nr8IPA4Ln0ogiA0GVBehxoIbcPNit-aKKU2n5dU7-p0voFX-zhf4oc_igruPUwY$, or unsubscribehttps://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AZGE7EAVSAGSHKU54GUJ2WTXAGZUVANCNFSM6AAAAAAWIW2AW4__;!!G2kpM7uM-TzIFchu!nr8IPA4Ln0ogiA0GVBehxoIbcPNit-aKKU2n5dU7-p0voFX-zhf4oc_iBMo9Ick$. You are receiving this because you were mentioned.Message ID: @.***>

[jinboxie]

Hi Qi @tangq , I've modified the input functions to pass the srf variables including "ch4_clim_srf, noy_clim_srf, n2o_clim_srf, o3_clim_srf" to xsfc in lin_strat_chem.F90. In this version, the ch4_avg_srf is also added and input as ch4max= linoz_ch4_avg_srf(1,1). The input function is read_zasrf_trc_linoz that has options of reading in ch4_avg_srf(time) or ch4_clim_srf(time,lat) or other surface variables.

[tangq]

@jinboxie , it seems that the new commit 5ae63d9 reverts many lines of changes (comments, nested loop order), which seems because you started from an old version. Can you open a new PR for this change because this PR has been merged? And revert the last commit on this PR?

[jinboxie]

Hi Qi @tangq ,I have updated the code. Ch4max is set to fixed value, and deleted the surface ch4_avg_srf time series input

[tangq]

@jinboxie , next step would be to complete the required tests following @wlin7 's instructions and document the results here. Thanks.

[jinboxie]

The current line records the new linv3 results using the new SSP forcing data from 1850-2100. The current input forcing data is different than linv3_1849-2015_2010JPL_cmip6_historical_10deg_58km_c20230405.nc in that the data are shifted 3 years later considering the transport time of tracers from surface to stratosphere. The current page documents the year 1850-1851 results using the new forcing and the old forcing. As below:

@jinboxie , the description says these figures are for year 1850-1851 results. Do you have post-2014 results using the SSP forcings?

A minor change: change the label of "percentage" to "%" and modify the colorbar accordingly, so it is clear what the relative numbers represent.

[jinboxie]

Hi Qi @tangq , please refer to the page for linoz_v3 using the new linv3 forcing data.

[tangq]

@jinboxie , the description says these figures are for year 1850-1851 results. Do you have post-2014 results using the SSP forcings?

A minor change: change the label of "percentage" to "%" and modify the colorbar accordingly, so it is clear what the relative numbers represent.

[jinboxie]

@jinboxie , the description says these figures are for year 1850-1851 results. Do you have post-2014 results using the SSP forcings?

A minor change: change the label of "percentage" to "%" and modify the colorbar accordingly, so it is clear what the relative numbers represent.

Hi Qi @tangq, it is revised accordingly with % and number for fourth picture *100 to reflect percentage number.

[jinboxie]

@jinboxie , the description says these figures are for year 1850-1851 results. Do you have post-2014 results using the SSP forcings? A minor change: change the label of "percentage" to "%" and modify the colorbar accordingly, so it is clear what the relative numbers represent.

Hi Qi @tangq, it is revised accordingly with % and number for fourth picture *100 to reflect percentage number.

Hi Qi @tangq . Please correct me if I have understand it wrong. The comparison above is between two algorithms and two forcing data. Orig is old linear interpolation algorithm+old forcing data (from 1849-2015, made by Juno), while new is new algorithm+ new forcing data. For post-2014 studies, we would have to use some future scenario forcing data, which Juno did not make. So we can't seem to have a comparable Original simulation for post-2014.

[tangq]

@jinboxie , for post-2014 period, we have compsets for SSP370 and SSP585 using linoz v2 as documented on this page. You will need to use the maint-2.0 code to run these two compsets as the reference cases for ozone, which is the only prognostic tracer in linoz v2. Does this make sense?

[jinboxie]

Hi Qi @tangq , plz refer to the figure below for comparison of the linoz-v2 between old (maint2.0 using linoz-v2 SSP forcing from Linoz v2 forcing) and new code base here (using linoz-v3 SSP forcing Linoz-v3 production) for 2020-2021. The simulations are ran under old: maint2.0, linoz-v2, using linoz-v2 SSP forcing for SSP370; new:current code base, linoz-v2, using linoz-v3 SSP forcing for SSP370; Another is the same as SSP370 except for SSP585

P.S. During our new simulations to open CMIP6 compset with linoz-v2 open, I experienced an ERROR because the call tropopause_e90_3d_output in components/eam/src/physics/cam/physpkg.F90 is error when E90 input if not set.I added a commit in this branch jinboxie/atm/chemUCI_amip_with_UCI-chem to update this small error for CMIP6 simulation with linoz-v2 only and no chemUCI. The statement in components/eam/src/chemistry/mozart/chemistry.F90 would also preclude our use of Linoz-v2 only (without chemUCI), so I currently take it off to no skip linoz.

  if (uci1_ndx <=0 ) then
    !   !write(iulog,*) 'skip Linoz, need to have tracer O3LNZ at least '
       write(iulog,*) 'skip Linoz, temporally change '
       do_lin_strat_chem = .false.
       return
    end if

[tangq]

P.S. During our new simulations to open CMIP6 compset with linoz-v2 open, I experienced an ERROR because the call tropopause_e90_3d_output in components/eam/src/physics/cam/physpkg.F90 is error when E90 input if not set.I added a commit in this branch jinboxie/atm/chemUCI_amip_with_UCI-chem to update this small error for CMIP6 simulation with linoz-v2 only and no chemUCI. The statement in components/eam/src/chemistry/mozart/chemistry.F90 would also preclude our use of Linoz-v2 only (without chemUCI), so I currently take it off to no skip linoz.

@jinboxie, can you post the links to the file changes of these additional commits? This simulation uses the master to run the v2 piControl, which runs linoz v2 without chemUCI and doesn't complain about e90. What are the differences?

[jinboxie]

Hi Qi @tangq . I made a new PR request for you to look at. The file changes is mostly in physkg.F90 (direct to the new PR). As you can see from line 2774,
call tropopause_e90_3d_output(state) This function is in NGD_v3atm's physpkg.F90, while in maint-2.0 it is not. This E90 is also in master, which should be used by Ziming. There are 3 scenarios for it

1. There's no tropopause_e90_3d_output, and no E90 input does not impact;
2. There's E90 input, although we do not use it for calculation;
3. There's no E90 input, and the function tropopause_e90_3d_output would look for it despite it does not need.

My use of maint-2.0 is 1, so no problem. Ziming's use of latest E3SM is most likely 2, which also works. My use of NGD_v3atm with linoz-v2 is 3 (as I was trying to use same SSP forcing of NGD_v3atm with maint-2.0, and thus no E90 input), so there's a small problem. So I had to add a judge statement to see if there's E90 or not. If no E90 input, we skip this call of tropopause_e90_3d_output.

[jinboxie]

Hi Qi @tangq , I talked with Ziming @keziming , and it is wuyin @wlin7 that made the change to fit for linoz-v2 simulations without E90 in master. His modifications are in components/eam/src/physics/cam/tropopause.F90 , where tropopause_e90_3d and tropopause_e90_3d_output have added checks for the existence of E90 input, one example is below:

subroutine tropopause_e90_3d_output(pstate)
    e90_ndx=-1
    e90_ndx = get_spc_ndx('E90')
    if (e90_ndx < 0) return
    call tropopause_e90_3d(pstate, tropLevB, tropLevU, tropFlag, tropFlagInt, tropP=tropP, tropT=tropT, tropZ=tropZ)

This is not added in v3atm tropopause.F90, and caused the problem.

[jinboxie]

Hi Qi @tangq ,I've made a confluence that sums up the info we have gathered here, please take a look at here, https://acme-climate.atlassian.net/wiki/spaces/ATMOS/pages/3788996686/Check+of+the+new+UCI+interpolation+and+forcing+data+for+Linoz