search

NOAA-EMC / UPP

Other
37 stars 100 forks source link

Refactor EMC_post decomposition from 1D to 2D as part of EMC_post refactoring #274

Closed GeorgeVandenberghe-NOAA closed 2 years ago

GeorgeVandenberghe-NOAA commented 3 years ago

EMC_post is currently decmposed on latitude (J) only. This is adequate for several more years but since post is generally being refactored, now is a good time to make the jump to 2D. A second goal is to make the 2D decomposition either flexible, or just have it mimic the ufs-weather-model decomposition so developers working on both codes can exploit commonality. This will be a modestly difficult project with most effort, figuring out the plumbing of the code (in progress). This issue is being created for management and project leader tracking and per EMC management directives and also best practices, results should be tracked through this Github issue or slack, NOT email.

There are many OTHER scaling issues in the post that are not affected by the decomposition. Most of the issues are orthogonal to the decomposition though and can be worked independently. The most salient is input I/O of model state fields in the standalone post.

By 03/01/2021:

HuiyaChuang-NOAA commented 3 years ago

@HuiyaChuang-NOAA @WenMeng-NOAA @GeorgeVandenberghe-NOAA @JesseMeng-NOAA - I also meant to comment. DTC's @kayeekayee is working on some development to translate all info in the itag to a formal fortran namelist (Issue #115) and has the work largely done and tested with current develop branch. Woudl this be of interest to review and push this feature enhancement to help aid the 2d decomp work?

@KateFriedman-NOAA @kayeekayee @GeorgeGayno-NOAA @JesseMeng-NOAA @WenMeng-NOAA have kayee started a PR to merge her changes in?

GeorgeVandenberghe-NOAA commented 3 years ago

Good for other reasons but I don't need it. Downward compatibility with current itag use would be helpful and I built that into my numx= itag read code. If we change itag to a namelist we will have to change runscripts to build it. Downward compatibility can be implemented if itag is retained as the input file but just with namelist stuff, by looking for non namelist stuff and branching to the old code in WRFPOST.f that handles it, otherwise going through the new (and improved of course) code that handles inputs as namelists.

On Fri, Oct 1, 2021 at 4:09 PM HuiyaChuang-NOAA @.***> wrote:

@HuiyaChuang-NOAA https://github.com/HuiyaChuang-NOAA @WenMeng-NOAA https://github.com/WenMeng-NOAA @GeorgeVandenberghe-NOAA https://github.com/GeorgeVandenberghe-NOAA @JesseMeng-NOAA https://github.com/JesseMeng-NOAA - I also meant to comment. DTC's @kayeekayee https://github.com/kayeekayee is working on some development to translate all info in the itag to a formal fortran namelist (Issue #115 https://github.com/NOAA-EMC/UPP/issues/115) and has the work largely done and tested with current develop branch. Woudl this be of interest to review and push this feature enhancement to help aid the 2d decomp work?

@KateFriedman-NOAA https://github.com/KateFriedman-NOAA @kayeekayee https://github.com/kayeekayee @GeorgeGayno-NOAA https://github.com/GeorgeGayno-NOAA @JesseMeng-NOAA https://github.com/JesseMeng-NOAA @WenMeng-NOAA https://github.com/WenMeng-NOAA have kayee started a PR to merge her changes in?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/UPP/issues/274#issuecomment-932518425, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FSIG2ZEPIPFVFFUXZTUEYIN7ANCNFSM4YVYMD5Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--

George W Vandenberghe

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

WenMeng-NOAA commented 3 years ago

@fossell and @kayeekeyee Could you start a PR to the branch develop so we may have further discussion for flexible namelists. Thanks!

KateFriedman-NOAA commented 3 years ago

@HuiyaChuang-NOAA Nothing into global-workflow yet but I'm happy to bring in a new UPP tag/version when the refactoring goes into UPP develop. Most of the workflow scripts for UPP are in the UPP so I won't have much to adjust on the global-workflow side for this other than just trying it in a cycled run when ready. Please open a new global-workflow issue with general info on the changes when ready to pass me this new UPP version and then I'll take it from there for the workflow side. Thanks!

WenMeng-NOAA commented 3 years ago

@KateFriedman-NOAA The UPP 2D decomposition from UPP refactor efforts has been developing yet. When this feature is merged in UPP develop branch, I will open issue and submit PR for the changes in global-workflow. Thanks!

fossell commented 3 years ago

@WenMeng-NOAA - Yes, @kayeekayee is planning to submit PR soon so we can all review proposed mods.

HuiyaChuang-NOAA commented 3 years ago

Good for other reasons but I don't need it. Downward compatibility with current itag use would be helpful and I built that into my numx= itag read code. If we change itag to a namelist we will have to change runscripts to build it. Downward compatibility can be implemented if itag is retained as the input file but just with namelist stuff, by looking for non namelist stuff and branching to the old code in WRFPOST.f that handles it, otherwise going through the new (and improved of course) code that handles inputs as namelists. … On Fri, Oct 1, 2021 at 4:09 PM HuiyaChuang-NOAA @.> wrote: @HuiyaChuang-NOAA https://github.com/HuiyaChuang-NOAA @WenMeng-NOAA https://github.com/WenMeng-NOAA @GeorgeVandenberghe-NOAA https://github.com/GeorgeVandenberghe-NOAA @JesseMeng-NOAA https://github.com/JesseMeng-NOAA - I also meant to comment. DTC's @kayeekayee https://github.com/kayeekayee is working on some development to translate all info in the itag to a formal fortran namelist (Issue #115 <#115>) and has the work largely done and tested with current develop branch. Woudl this be of interest to review and push this feature enhancement to help aid the 2d decomp work? @KateFriedman-NOAA https://github.com/KateFriedman-NOAA @kayeekayee https://github.com/kayeekayee @GeorgeGayno-NOAA https://github.com/GeorgeGayno-NOAA @JesseMeng-NOAA https://github.com/JesseMeng-NOAA @WenMeng-NOAA https://github.com/WenMeng-NOAA have kayee started a PR to merge her changes in? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#274 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FSIG2ZEPIPFVFFUXZTUEYIN7ANCNFSM4YVYMD5Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. -- George W Vandenberghe IMSG at NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 @. 301-683-3769(work) 3017751547(cell)

If NUMX is read in as a variable in namelist, then it can be an optional variable too.
I looked at what you did and it looked fine. We can talk at next week's 2D decomposition tagup to see if we want to change numx to be in namelist

HuiyaChuang-NOAA commented 3 years ago

@HuiyaChuang-NOAA Nothing into global-workflow yet but I'm happy to bring in a new UPP tag/version when the refactoring goes into UPP develop. Most of the workflow scripts for UPP are in the UPP so I won't have much to adjust on the global-workflow side for this other than just trying it in a cycled run when ready. Please open a new global-workflow issue with general info on the changes when ready to pass me this new UPP version and then I'll take it from there for the workflow side. Thanks!

As Wen mentioned, it's still work in progress. The projected completion date is June next year.

HuiyaChuang-NOAA commented 3 years ago

@JesseMeng-NOAA @BoCui-NOAA @GeorgeVandenberghe-NOAA @WenMeng-NOAA Spoke to Wen and DTC submitted a PR #402 for making entire itag a namelist. Wen is running regression tests for this PR. She estimated this PR will be merged in 3 weeks if all goes well. Wen will then work with Jesse to sync this merge from develp to 2D decomposition branch, at which time, we can make numx part of namelist as well. I will discuss this at next week's tagup.

GeorgeVandenberghe-NOAA commented 3 years ago

Will it still work with the old itag or will that have to be changed?

On Thu, Oct 7, 2021 at 1:43 PM HuiyaChuang-NOAA @.***> wrote:

@JesseMeng-NOAA https://github.com/JesseMeng-NOAA @BoCui-NOAA https://github.com/BoCui-NOAA @GeorgeVandenberghe-NOAA https://github.com/GeorgeVandenberghe-NOAA @WenMeng-NOAA https://github.com/WenMeng-NOAA Spoke to Wen and DTC submitted a PR #402 https://github.com/NOAA-EMC/UPP/pull/402 for making entire itag a namelist. Wen is running regression tests for this PR. She estimated this PR will be merged in 3 weeks if all goes well. Wen will then work with Jesse to sync this merge from develp to 2D decomposition branch, at which time, we can make numx part of namelist as well. I will discuss this at next week's tagup.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/UPP/issues/274#issuecomment-938015119, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FW45C7SBKHWDH32AITUFXL4JANCNFSM4YVYMD5Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--

George W Vandenberghe

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

WenMeng-NOAA commented 3 years ago

@GeorgeVandenberghe-NOAA There will be format changes in old tag with all in namelist. Right now, you may modify itag for 2d decomposition testing. I will work with you for updating itag later.

JesseMeng-NOAA commented 3 years ago

@GeorgeVandenberghe-NOAA Hi George, I am modifying the subroutine MDL2THANDPV.f to 2D decomposition. This is the first subroutine that I test so far that actually does i-direction differential. EXCH.f, has the following lines of ibl=max(ista-1,1) ibu=min(im,iend+1) that set the i boundaries to 1 and im. If I understand it correctly, at i=1 the left halo carries the same value as i=1. So does at i=im, the right halo carries the same value as i=im. This works fine with 1D decomposition since the arrays have access to the entire length of 1:im and can find the left neighbor of i=1 to be im and the right neighbor of i=im to be 1. But in 2D decomposition, at i=1 it does not have access to the left neighbor of im; and similar issue with i=im. Could you help to revisit this? Thanks! -Jesse

GeorgeVandenberghe-NOAA commented 3 years ago

Exch exchanges neighbor columns so thus should just already work

On Thursday, October 21, 2021, Jesse Meng @.***> wrote:

@GeorgeVandenberghe-NOAA Hi George, I am modifying the subroutine MDL2THANDPV.f to 2D decomposition. This is the first subroutine that I test so far that actually does i-direction differential. EXCH.f, has the following lines of ibl=max(ista-1,1) ibu=min(im,iend+1) that set the i boundaries to 1 and im. If I understand it correctly, at i=1 the left halo carries the same value as i=1. So does at i=im, the right halo carries the same value as i=im. This works fine with 1D decomposition since the arrays have access to the entire length of 1:im and can find the left neighbor of i=1 to be im and the right neighbor of i=im to be 1. But in 2D decomposition, at i=1 it does not have access to the left neighbor of im; and similar issue with i=im. Could you help to revisit this? Thanks! -Jesse

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. < https://ci3.googleusercontent.com/proxy/JA0fuDEtYB0-gjECuBgymPzcZqMbqSQw0I2kY1FY52ooH9m97sUAW1YyVJpypfbGitL67eK5C1rQSUOxQW-4pXv6f3wlWCt5w8hDv-Sr72UGLNnDQDQX0LvcSTPglYiiMoKAzUmtCdl_Zuqdid8YSjoIveXfxVVKYhHxhSdtUr2pqfoCAXlAkNpDBxARhDcr7fYhi4gZCR1d68knN_6cAK-mL7K6qv1xFKzjKzmAYw=s0-d-e1-ft#https://github.com/notifications/beacon/ANDS4FU2D4GUJPGV3B262ADUICG6BA5CNFSM4YVYMD52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOHCIUYDY.gif

--

George W Vandenberghe

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

GeorgeVandenberghe-NOAA commented 3 years ago

We need to think about this. Using the im value to replace the 0 value only works for global domain and that was also the case for the 1ď decomposition. On Thursday, October 21, 2021, Jesse Meng @.***> wrote:

@GeorgeVandenberghe-NOAA Hi George, I am modifying the subroutine MDL2THANDPV.f to 2D decomposition. This is the first subroutine that I test so far that actually does i-direction differential. EXCH.f, has the following lines of ibl=max(ista-1,1) ibu=min(im,iend+1) that set the i boundaries to 1 and im. If I understand it correctly, at i=1 the left halo carries the same value as i=1. So does at i=im, the right halo carries the same value as i=im. This works fine with 1D decomposition since the arrays have access to the entire length of 1:im and can find the left neighbor of i=1 to be im and the right neighbor of i=im to be 1. But in 2D decomposition, at i=1 it does not have access to the left neighbor of im; and similar issue with i=im. Could you help to revisit this? Thanks! -Jesse

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. < https://ci3.googleusercontent.com/proxy/JA0fuDEtYB0-gjECuBgymPzcZqMbqSQw0I2kY1FY52ooH9m97sUAW1YyVJpypfbGitL67eK5C1rQSUOxQW-4pXv6f3wlWCt5w8hDv-Sr72UGLNnDQDQX0LvcSTPglYiiMoKAzUmtCdl_Zuqdid8YSjoIveXfxVVKYhHxhSdtUr2pqfoCAXlAkNpDBxARhDcr7fYhi4gZCR1d68knN_6cAK-mL7K6qv1xFKzjKzmAYw=s0-d-e1-ft#https://github.com/notifications/beacon/ANDS4FU2D4GUJPGV3B262ADUICG6BA5CNFSM4YVYMD52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOHCIUYDY.gif

--

George W Vandenberghe

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

JesseMeng-NOAA commented 3 years ago

@HuiyaChuang-NOAA @WenMeng-NOAA Several things with MDL2THANDPV.f that we need to discuss.

  1. It looks like the current EXCH.f probably does not cross i=1 and i=im. I am seeking George's help.
  2. Line 68: imb2 = im /2 and later on in the 206-216 block it does calculations with 2 points from the opposite location around the i circle. This cannot be done in x decomposition.
  3. Line 757: call poleavg( ) This calculates averages around the i circle at j=1 and j=jm. This cannot be done in x decomposition.
GeorgeVandenberghe-NOAA commented 3 years ago

All of this is tractable. Lets think about approaches.

On Thursday, October 21, 2021, Jesse Meng @.***> wrote:

@HuiyaChuang-NOAA @WenMeng-NOAA Several things with MDL2THANDPV.f that we need to discuss.

It looks like the current EXCH.f probably does not cross i=1 and i=im. I am seeking George's help. Line 68: imb2 = im /2 and later on in the 206-216 block it does calculations with 2 points from the opposite location around the i circle. This cannot be done in x decomposition. Line 757: call poleavg( ) This calculates averages around the i circle at j=1 and j=jm. This cannot be done in x decomposition.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. < https://ci4.googleusercontent.com/proxy/1O-M3XlkdNawr7mVU0WPXtnT3xb3KM290oLuKUuz0lALycZzzCI_XtPD3jI0hc0uv2ZL3uMiKG6YR-Gt6FH7cYfwtfh739ztqnEMR1inzXtVVHXY1VhKA_lm1REUDrotq_DW3cc_NxShTVxTVkYglF5vfSTm40k5SZaTlkDxWgg3olWloED5bZCzmO28Vq-kSyH3job1goXRwlm0yJCvsEqJtWPNbHQMLBFLQWKYmA=s0-d-e1-ft#https://github.com/notifications/beacon/ANDS4FTQDGCLZMGVJTA5LALUICMKTA5CNFSM4YVYMD52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOHCIZRRY.gif

--

George W Vandenberghe

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

JesseMeng-NOAA commented 3 years ago

Confirmed that my 2D decomposition tests, for both numx=1 and numx=12, MDL2THANDPV.f generated potential vorticity field are mostly identical to that from dev branch, except for the 4 lines of i=1,i=im, j=1, j=jm.

GeorgeVandenberghe-NOAA commented 3 years ago

There should be no difference with numx=1 since that morphs to the 1D decomposition.

We need to think about what to do for boundaries esp. if the domain isn't global.

On Fri, Oct 22, 2021 at 9:38 AM Jesse Meng @.***> wrote:

Confirmed that my 2D decomposition tests, for both numx=1 and numx=12, MDL2THANDPV.f generated potential vorticity field are mostly identical to that from dev branch, except for the 4 lines of i=1,i=im, j=1, j=jm.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/UPP/issues/274#issuecomment-949640294, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FWQP2MPQ2WXTZ2PZADUIFSLZANCNFSM4YVYMD5Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--

George W Vandenberghe

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

JesseMeng-NOAA commented 3 years ago

Yes numx=1 has no difference. Sorry. numx=12 was different only in the four lines of x=1, x=im, y=1, y=jm.

HuiyaChuang-NOAA commented 3 years ago

@JesseMeng-NOAA @GeorgeGayno-NOAA Thank you. Let me think about a way to deal with vorticity computation at the boundaries. Let's discuss this at next Tuesday's 2D decomposition tag up

GeorgeGayno-NOAA commented 3 years ago

@JesseMeng-NOAA @GeorgeVandenberghe-NOAA Thank you. Let me think about a way to deal with vorticity computation at the boundaries. Let's discuss this at next Tuesday's 2D decomposition tag up

Do you mean @GeorgeVandenberghe-NOAA? I don't work with the UPP. But somehow I am getting updates for this issue.

HuiyaChuang-NOAA commented 3 years ago

@JesseMeng-NOAA @GeorgeVandenberghe-NOAA Thank you. Let me think about a way to deal with vorticity computation at the boundaries. Let's discuss this at next Tuesday's 2D decomposition tag up

Do you mean @GeorgeVandenberghe-NOAA? I don't work with the UPP. But somehow I am getting updates for this issue.

@GeorgeGayno-NOAA yes. thanks for letting me know I used wrong George.

GeorgeVandenberghe-NOAA commented 3 years ago

How do we do it now for regional grids where the I domain does not span all longitudes? Treating a(im) and a(1) as adjacent produces a smooth derivative for a global domain because they are adjacent. But this is not the case for a regional grid. We deal with regional grids now, so what do we do?

On Fri, Oct 22, 2021 at 1:55 PM HuiyaChuang-NOAA @.***> wrote:

@JesseMeng-NOAA https://github.com/JesseMeng-NOAA @GeorgeGayno-NOAA https://github.com/GeorgeGayno-NOAA Thank you. Let me think about a way to deal with vorticity computation at the boundaries. Let's discuss this at next Tuesday's 2D decomposition tag up

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/UPP/issues/274#issuecomment-949847255, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FSRAZEA7SQ254OOKZ3UIGQRLANCNFSM4YVYMD5Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--

George W Vandenberghe

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

HuiyaChuang-NOAA commented 3 years ago

@JesseMeng-NOAA @GeorgeVandenberghe-NOAA in lines 204-216 in MDL2THANDPV.f, what UPP was doing was computing vorticity at J=1 by using centered differencing across the South Pole or J=JM across North Pole. This used to work when I is not decomposed. My thought is we will have to switch to backward differencing. Also, @JesseMeng-NOAA we have more than 1 places where vorticity is computed. I think it would be good to add a subroutine to function to compute vorticity in UPP_PHYSICS.f

JesseMeng-NOAA commented 3 years ago

One more thing I have mentioned in my comment above but not in the conversation today. There is also the subroutine POLEAVG.f that computes averages around the entire i circle at j=1 and j=jm. @GeorgeVandenberghe-NOAA could you please also look into this? Thanks!

WenMeng-NOAA commented 3 years ago

@GeorgeVandenberghe-NOAA The itag for gfs is created in post workflow script. You might see new gfs itag format like: &model_inputs fileName='nemsfile' IOFORM='netcdfpara' grib='grib2' DateStr='2021-02-16_06:00:00' MODELNAME='GFS' fileNameFlux='flxfile' / &NAMPGB KPO=57,PO=1000.,975.,950.,925.,900.,875.,850.,825.,800.,775.,750.,725.,700.,675.,650.,625.,600.,575.,550.,525.,500.,475.,450.,425.,400.,375.,350.,325.,300.,275.,250.,225.,200.,175.,150.,125.,100.,70.,50.,40.,30.,20.,15.,10.,7.,5.,3.,2.,1.,0.7,0.4,0.2,0.1,0.07,0.04,0.02,0.01, /

JesseMeng-NOAA commented 3 years ago

Done all my shares of 2D DECOMPOSITION subroutines including those I took over from Bo. Wen's regression test in venus:/u/Wen.Meng/noscrubd/ncep_post/post_regression_test_new still works and I added numx=1 in itag namelist. numx=1 passes regression test for most of the models except for nmmb RH on just a few sigma levels, not all levels. I will look into this. 1125:462230579:RH:0.47-1 sigma layer:rpn_corr=0.992464:rpn_rms=2.55337 1126:462562119:RH:0.47-0.96 sigma layer:rpn_corr=0.993236:rpn_rms=2.52673 1127:462897214:RH:0.18-0.47 sigma layer:rpn_corr=0.996433:rpn_rms=1.58959 1128:463197281:RH:0.84-0.98 sigma layer:rpn_corr=0.980175:rpn_rms=3.61572 1129:463563671:MCONV:0.85-1 sigma layer:rpn_corr=0.999514:rpn_rms=6.62663e-09 1482:487727642:RH:0.47-1 sigma layer:rpn_corr=0.992464:rpn_rms=2.55337

BoCui-NOAA commented 3 years ago

Thanks Jesse' support and took over many subroutines' 2d decomposition. Now I am also done with all my tasks. I ran Wen's regression test with numx=1, the regression test worked for most of the models except for nmmb RH on some sigma levels, the similar results as Jesse's.

Bo

On Wed, Nov 3, 2021 at 9:45 AM Jesse Meng @.***> wrote:

Done all my shares of 2D DECOMPOSITION subroutines including those I took over from Bo. Wen's regression test in venus:/u/Wen.Meng/noscrubd/ncep_post/post_regression_test_new still works and I added numx=1 in itag namelist. numx=1 passes regression test for most of the models except for nmmb RH on just a few sigma levels, not all levels. I will look into this. 1125:462230579:RH:0.47-1 sigma layer:rpn_corr=0.992464:rpn_rms=2.55337 1126:462562119:RH:0.47-0.96 sigma layer:rpn_corr=0.993236:rpn_rms=2.52673 1127:462897214:RH:0.18-0.47 sigma layer:rpn_corr=0.996433:rpn_rms=1.58959 1128:463197281:RH:0.84-0.98 sigma layer:rpn_corr=0.980175:rpn_rms=3.61572 1129:463563671:MCONV:0.85-1 sigma layer:rpn_corr=0.999514:rpn_rms=6.62663e-09 1482:487727642:RH:0.47-1 sigma layer:rpn_corr=0.992464:rpn_rms=2.55337

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/UPP/issues/274#issuecomment-959115734, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMYNKUDPT7EBHYFK6AS4VR3UKFDJHANCNFSM4YVYMD5Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--

Bo Cui

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2063

College Park, MD 20740

@.***

301-683-3710

GeorgeVandenberghe-NOAA commented 2 years ago

So.ooo How do we want to handle that IMx2 structure that is needed for full polar I domains. Should we change the exch binding or just document something extra that is available in anything that uses CTLBLK_MOD.f after exch is called, or generate an exch_pole routine that fills this structure when called with the subdomain field, similar to what exch does with the boundaries except what's returned is a new filled data structure. None of this is hard.

Note exch.f itself remains full of debug code that should be removed as soon as the 2D decomposition is debugged for all domains and all reasonable decompositions.

GeorgeVandenberghe-NOAA commented 2 years ago

I have run into a snag with the boundary halos. The subdomains are dimensioned with the lowest I to be the greater of ista-2, or 1. The 1 is causing issues for I=1, no place to put the cyclic IM value to the left. An analagous situation occurs for I=im.

Pondering how to cleanly fix this.

HuiyaChuang-NOAA commented 2 years ago

Summary from 11/23/2021 2D decomposition meeting today:

BoCui-NOAA commented 2 years ago

@JesseMeng-NOAA Please check the following two routines and lines and see if (i,j,jj) would be replaced by (i,ii,j,jj).

MISCLN.f:4177:! $omp parallel do private(i,j,jj)

SURFCE.f:463:!$omp parallel do private(i,j,jj) SURFCE.f:913:!$omp parallel do private(i,j,jj) SURFCE.f:1738:!$omp parallel do private(i,j,jj) SURFCE.f:1966:!$omp parallel do private(i,j,jj) SURFCE.f:2334:!$omp parallel do private(i,j,jj) SURFCE.f:2607:!$omp parallel do private(i,j,jj) SURFCE.f:2724:!$omp parallel do private(i,j,jj) SURFCE.f:6083:!$omp parallel do private(i,j,jj) SURFCE.f:463:!$omp parallel do private(i,j,jj) SURFCE.f:913:!$omp parallel do private(i,j,jj) SURFCE.f:1738:!$omp parallel do private(i,j,jj) SURFCE.f:1966:!$omp parallel do private(i,j,jj) SURFCE.f:2334:!$omp parallel do private(i,j,jj) SURFCE.f:2607:!$omp parallel do private(i,j,jj) SURFCE.f:2724:!$omp parallel do private(i,j,jj) SURFCE.f:6083:!$omp parallel do private(i,j,jj)

JesseMeng-NOAA commented 2 years ago

Yes all should be (i,ii,j,jj). Thanks for double checking. I will update those.

@JesseMeng-NOAA Please check the following two routines and lines and see if (i,j,jj) would be replaced by (i,ii,j,jj).

MISCLN.f:4177:! $omp parallel do private(i,j,jj)

SURFCE.f:463:!$omp parallel do private(i,j,jj) SURFCE.f:913:!$omp parallel do private(i,j,jj) SURFCE.f:1738:!$omp parallel do private(i,j,jj) SURFCE.f:1966:!$omp parallel do private(i,j,jj) SURFCE.f:2334:!$omp parallel do private(i,j,jj) SURFCE.f:2607:!$omp parallel do private(i,j,jj) SURFCE.f:2724:!$omp parallel do private(i,j,jj) SURFCE.f:6083:!$omp parallel do private(i,j,jj) SURFCE.f:463:!$omp parallel do private(i,j,jj) SURFCE.f:913:!$omp parallel do private(i,j,jj) SURFCE.f:1738:!$omp parallel do private(i,j,jj) SURFCE.f:1966:!$omp parallel do private(i,j,jj) SURFCE.f:2334:!$omp parallel do private(i,j,jj) SURFCE.f:2607:!$omp parallel do private(i,j,jj) SURFCE.f:2724:!$omp parallel do private(i,j,jj) SURFCE.f:6083:!$omp parallel do private(i,j,jj)

HuiyaChuang-NOAA commented 2 years ago

@JesseMeng-NOAA @BoCui-NOAA @GeorgeVandenberghe-NOAA @WenMeng-NOAA @fossell Summary from 12/21/2021 2D decomposition tag-up:

  1. Jesse reported the team is done with 2D decomposition work, thanks to George's recent work to update EXCH to carry our derivatives across the poles.
  2. Jesse updated UPP so only GFS runs with 2D decomposition.
  3. Wen is planning to unify initialization for GFS and RRFS when using parallel NetCDF read. When that's completed, both RRFS and GFS can run with 2D decomposition. However, she needs someone from modeling side to unify diag table first.
  4. Jesse and Bo both ran UPP regression tests with numx=1 for Y decomposing only and the 2D decomposition branch re-produced all regression test output. They also ran with numx>1 for GFS and confirmed output reproduced as well.

Action items:

  1. George will work on cleaning up EXCH.f and document his work
  2. Jesse and Bo will work on a user guide on how to run UPP with 2D decomposition
  3. Kate mentioned Tracy is working on adding Doxygen to UPP and can work with George to document his work
  4. Huiya will add to agenda for 1/4/2022 2D decomposition meeting to discuss release plan for 2D decomposed UPP
JesseMeng-NOAA commented 2 years ago

Tested George's fix for EXCH.f and identical results were reproduced. Code pushed to github. A developer's guide for testing UPP GFS 2D Decomposition has been drafted, https://docs.google.com/document/d/1jvky8T3S7uo7z_Y_hHDj9Ak0DgmkPI75Q8RcqAazeno

HuiyaChuang-NOAA commented 2 years ago

Summary from 1/4/2022 decomposition meeting: . DTC will run their version of EMC's version's UPP regression tests on 2D decomposition branch and report their findings at next tag-up . George will do a quick clean-up and hand off his code to Jesse to commit (done) . Jesse will work on a quick user guide on running UPP with 2D decomposition (done) . Wen will work on syncing 2D decomposition branch to develop . George will continue to clean up and document. . Wen would like 2D decomposition branch to be tested on Jet and Orion as GSL runs on Jet and HAFS runs on Orion. The plan is: a) Wen will provide Jesse and Bo a script to run UPP on Orion. b) Jesse and Bo will test 2D decomposition branch on Orion. c) After DTC verified their testing is positive, Huiya will reach out to GSL to test 2D decomposition branch on Jet. . The group discussed areas where EPIC infrastructure team can help enhance UPP: a) Short-term: updating inline post to use and test 2D decomposed UPP and unification of RRFS and GFS UPP interfaces b) Long term: adding bufr sounding to inline post

HuiyaChuang-NOAA commented 2 years ago

Summary from 1/18 2D decomposition tag-up

  1. George cleaned up EXCH and MPI_FIRST subroutines and will send them to Bo and Jesse to run UPP regression tests with.
  2. Bo and Jesse will run the above-mentioned tests and record timing and will send their findings to the group.
  3. DTC tested 2D decomposition branch in their own regression tests and received bit identical results. However, since they only use serial NetCDF read for GFS, their tests only involve Y only decomposition.
  4. DTC is working on adding GFS parallel NetCDF read into their regression tests and will update EMC on their progress later this week.
  5. George will continue to work on documentaion
  6. Huiya will email Jun about possibility of her team testing 2D decomposition branch in in-line post.
HuiyaChuang-NOAA commented 2 years ago

Summary from 2/15 2D decomposition meeting:

  1. Jesse and Bo have been updating UPP inline interfaces to do 2D decomposition. They encountered issues running inline post using 2D decomposition branch with Y decomposition only. They're getting Jun's help to debug.
  2. George was working on wcoss2 issues.
JesseMeng-NOAA commented 2 years ago

The interface between UPP/post_2d_decomp and ufs-weather-model/FV3 inline post has been developed for both gfs and regional models.

To test this functionality, First build the regression test bases with the ufs-weather-model/develop branch by running ufs-weather-model/tests/rt.sh with only the two controls defined in rt.conf control regional_control

Then run the UPP/post_2d_decomp regression test. Under the ufs-weather-model test directory, checkout Jesse's fv3atm/upp_2d_decomp branch https://github.com/JesseMeng-NOAA/fv3atm/tree/upp_2d_decomp and Wen's UPP/upp_2d_decomp branch https://github.com/WenMeng-NOAA/UPP/tree/post_2d_decomp

Turn on the WRITE_DOPOST flag in ufs-weather-model/tests/tests/control_2dwrtdecomp ufs-weather-model/tests/tests/regional_control_2dwrtdecomp export WRITE_DOPOST=.true. and run ufs-weather-model/tests/rt.sh with only the two 2dwrtdecomp experiments defined in rt.conf control_2dwrtdecomp regional_control_2dwrtdecomp

More details can be found in this menu, https://docs.google.com/document/d/15whXaBCQzUmJUJa1TBCTrFUsZYpebSmqVwexbFcAfaw/edit

GeorgeVandenberghe-NOAA commented 2 years ago 
I started the documentation after breaking from continuing WCOSS

crises. Below is a very high level schematic. Information on the actual variables changed will follow later.

! The 1D decomposition can read state from a model forecast file, either by reading on rank 0 ! and scattering, or by doing MPI_IO on the model history file using either nemsio, sigio, ! or netcdf serial or parallel I/O. Very old ! post tags also implement the more primitive full state broadcast or (a ! performance bug rectified 10/17) read the entire state on all tasks. This ! is mentioned in case a very old tag is encountered. The 2D decomposition ! only supports MPI_IO for the general 2D case but all I/O methods remain ! supported for the 1D special case of the 2D code. This 1D special case ! works for all cases currently supported by older 1D tags and branches. ! ! to repeat, ONLY 2D NETCDF PARALLEL I/O WILL BE SUPPORTED FOR THE ! GENERAL CASE OF 2D DECOMPOSITION. ! ! **** 2D design enhancements

! ! ! ! The 2D decomposition operates on subdomains with some latitudes and ! some longitudes. The subdomains are lonlat rectangles rather than ! strips. This means state must be chopped into pieces in any ! scatter operation and the pieces reassembled in any gather ! operation that requires a continuous in memory state. I/O and halo ! exchanges both require significantly more bookkeeping. ! ! ! The structural changes needed for the 2D decomposition are ! implemented in MPI_FIRST.f and CTLBLK.f! CTLBLK.f contains ! numerous additional variables describing left and right domain ! boundaries. Many additional changes are also implemented in EXCH.f ! to support 2D halos. Many additional routines required addition of ! the longitude subdomain limits but changes to the layouts are ! handled in CTLBLK.f and the "many additional routines" do not ! require additional changes when subdomain shapes are changed and ! have not been a trouble point. ! ! ! Both MPI_FIRST and EXCH.f contain significant additional test code ! to exchange arrays containing grid coordinates and ensure EXACT ! matches for all exchanges before the domain exchanges are ! performed. This is intended to trap errors in the larger variety of ! 2D decomposition layouts that are possible and most of it can ! eventually be removed or made conditional at build and run time. ! !

On Tue, Mar 1, 2022 at 8:57 AM WenMeng-NOAA @.***> wrote:

Assigned #274 https://github.com/NOAA-EMC/UPP/issues/274 to @GeorgeVandenberghe-NOAA https://github.com/GeorgeVandenberghe-NOAA.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/UPP/issues/274#event-6162231324, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FTM3EGKKBARI6RLLCDU5YO5NANCNFSM4YVYMD5Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were assigned.Message ID: @.***>

--

George W Vandenberghe

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

HuiyaChuang-NOAA commented 2 years ago

I started the documentation after breaking from continuing WCOSS crises. Below is a very high level schematic. Information on the actual variables changed will follow later. ! The 1D decomposition can read state from a model forecast file, either by reading on rank 0 ! and scattering, or by doing MPI_IO on the model history file using either nemsio, sigio, ! or netcdf serial or parallel I/O. Very old ! post tags also implement the more primitive full state broadcast or (a ! performance bug rectified 10/17) read the entire state on all tasks. This ! is mentioned in case a very old tag is encountered. The 2D decomposition ! only supports MPI_IO for the general 2D case but all I/O methods remain ! supported for the 1D special case of the 2D code. This 1D special case ! works for all cases currently supported by older 1D tags and branches. ! ! to repeat, ONLY 2D NETCDF PARALLEL I/O WILL BE SUPPORTED FOR THE ! GENERAL CASE OF 2D DECOMPOSITION. ! ! **** 2D design enhancements **** ! ! ! ! The 2D decomposition operates on subdomains with some latitudes and ! some longitudes. The subdomains are lonlat rectangles rather than ! strips. This means state must be chopped into pieces in any ! scatter operation and the pieces reassembled in any gather ! operation that requires a continuous in memory state. I/O and halo ! exchanges both require significantly more bookkeeping. ! ! ! The structural changes needed for the 2D decomposition are ! implemented in MPI_FIRST.f and CTLBLK.f! CTLBLK.f contains ! numerous additional variables describing left and right domain ! boundaries. Many additional changes are also implemented in EXCH.f ! to support 2D halos. Many additional routines required addition of ! the longitude subdomain limits but changes to the layouts are ! handled in CTLBLK.f and the "many additional routines" do not ! require additional changes when subdomain shapes are changed and ! have not been a trouble point. ! ! ! Both MPI_FIRST and EXCH.f contain significant additional test code ! to exchange arrays containing grid coordinates and ensure EXACT ! matches for all exchanges before the domain exchanges are ! performed. This is intended to trap errors in the larger variety of ! 2D decomposition layouts that are possible and most of it can ! eventually be removed or made conditional at build and run time. ! ! … On Tue, Mar 1, 2022 at 8:57 AM WenMeng-NOAA @.> wrote: Assigned #274 <#274> to @GeorgeVandenberghe-NOAA https://github.com/GeorgeVandenberghe-NOAA. — Reply to this email directly, view it on GitHub <#274 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FTM3EGKKBARI6RLLCDU5YO5NANCNFSM4YVYMD5Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were assigned.Message ID: @.> -- George W Vandenberghe IMSG at NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 @.*** 301-683-3769(work) 3017751547(cell)

Thank you, George. This high level documentation looks good. Looking forward to your documentation on variables you added/updated and their descriptions

GeorgeVandenberghe-NOAA commented 2 years ago

Additional post documentation. This is NOT in any routines but is a separate little document (so far)

! the following is found in CTLBLK.f and shared in the rest of UPP through use of CTLBLK.mod

! im. integer full longitude domain ! jm integer full latitude domain ! ! jsta integer start latitude on a task subdomain ! jend integer end latitude on a task subdomain ! ista integer start longitude on a task subdomain ! iend integer end longitude on a task subdomain ! ista_2l integer start longitude -2 of the subdomain ! iend_2u integer end longitude +2 of the subdomain ! jsta_2l integer start latitude -2 of the subdomain ! jend_2u integer end latitude +2 of the subdomain ! The shape of the subdomain is ista_2l:iend_2u,jsta_2l:jend_2u so it includes the halos although the halos ! are not populated until exhange is done in EXCH.f ! ! because of halos we need more bounds defined ! ! jsta_m single latitude below begin latitude of subdomain. ! jend_m single latitude above end latitude of subdomain ! jsta_m2 second latitude below begin latitude of subdomain . Apparently not used currently in compuations but subdomain shape uses this ! jend_m2 second latitude above end latitude of subdomain. apparently not used currently but subdomain shape uses this ! ! ista_m single longitude before begin longitude ! iend_m single longitude after end longitude ! ista_m2 second longitude before begin longitude ! iend_m2 second longitude after end longitude

! ileft. MPI rank containing the last longitude before ista_m ! iright. MPI rank containing the first longitude after iend_m ! iup MPI rank containing the first latitude after jend ! idn MPI rank containing the last latitude before jsta

! ileftb. MPI rank containing the last longitude before ista_m but for cyclic boundary conditions where "last" at the beginning ! is the other end of the domain (apparently unused and replaced with local calculation) ! irightb. MPI rank containing the first longitude after iend_m but for cyclic boundary conditions where "first" at the beginning ! is the other end of the domain (apparently unused and replaced with local calculation)

On Tue, Mar 1, 2022 at 8:57 AM WenMeng-NOAA @.***> wrote:

Assigned #274 https://github.com/NOAA-EMC/UPP/issues/274 to @GeorgeVandenberghe-NOAA https://github.com/GeorgeVandenberghe-NOAA.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/UPP/issues/274#event-6162231324, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FTM3EGKKBARI6RLLCDU5YO5NANCNFSM4YVYMD5Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were assigned.Message ID: @.***>

--

George W Vandenberghe

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

fossell commented 2 years ago

@GeorgeVandenberghe-NOAA - Is this second piece of documentation you posted going to go into the same overview document as the previous schematic text you provided? I'm mocking up some pages to be included in the formal documentation, so just wanted to check.

GeorgeVandenberghe-NOAA commented 2 years ago

Yes. Its in the overview document. Not an individual routine docblock.

On Monday, March 14, 2022, Kate Fossell @.***> wrote:

@GeorgeVandenberghe-NOAA - Is this second piece of documentation you posted going to go into the same overview document as the previous schematic text you provided? I'm mocking up some pages to be included in the formal documentation, so just wanted to check.

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.< https://ci6.googleusercontent.com/proxy/MLW8Tp1MYLv0BSeztVweDxOnQG5R2Z928xx9grkXUJUVTM4iySo1rg9XdMrsJGkDpX6-V-_yN9ZssMy44GRkOOS9Ik_ek5FUoTfGedHXxbh8yK_boRaUphA7zWTPUg1ROxPvtvQAWyKbw-wYscCHbSQxbzaFuDLRRwzQqKoMi53MOlmmQw_-iaDmkM3XrRnoE-eyt66Mlw_RaGhAGJ2s9cYHPQKwY-g3RIMfwAmn3Q=s0-d-e1-ft#https://github.com/notifications/beacon/ANDS4FVAXXCN2CVQHLEPT23U77VKBA5CNFSM4YVYMD52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOH6QHWDQ.gif>Message ID: @.***>

--

George W Vandenberghe

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

HuiyaChuang-NOAA commented 2 years ago

Summary from 3/15/2022 2D decomposition meeting:

  1. Bo ran some more tests with 2D decomposition branch with extreme case scenario such as no y decomposition and post failed. Jesse added condition to reset numx to 1 if numx>num_proc/2
  2. Jesse fixed issues Bo discover. He also opened an issue and submitted a PR on UFS Github. He's waiting for Jun to let him know what's next.
  3. Wen said Jun will ask people to test Jesse's PR. She ran Jesse's UFS PR but did not generate bit identical results. Jesse will help her figure out why.
  4. George is working on documenting variables he added to do 2D decomposition.
  5. Kate will help George use doxygen to add George's documentaion.
  6. George would like someone not as familiar with 2D decomposition code to read his documentation. Huiya volunteered.
  7. Huiya will email Jun to get timeline on when her team will test Jesse's PR as UPP re-engineering needs to finish by the end of May.
  8. Huiya will ask Shelley to add UPP re-engineering tasks for Jesse and Bo for Q3FY22
WenMeng-NOAA commented 2 years ago

@HuiyaChuang-NOAA Since Jesse's UFS PR is available on UFS repository, you might invite model developers from GFS, RRFS, and HAFS to test this upp 2d decomposition capability. @junwang-noaa may chime in.

WenMeng-NOAA commented 2 years ago

The UPP standalone regression tests failed at RRFS and HAFS due to syncing PR #441. The regional FV3 read interface INITPOST_NETCDF.f wasn't updated with 2D decomposition. The workaround is turning off RRFS and HAFS in your UPP RT tests. I would fix the issue after PR #453 committed (Unify regional and global FV3 read interfaces).

HuiyaChuang-NOAA commented 2 years ago

Summary from 3/29/2022 meeting:

  1. Wen is planning to commit her PR to unify stand-alone UPP interfaces to develop this or next week.
  2. Bo will then merge this change to 2D decomposition branch and resolve the conflicts.
  3. Jesse will continue to resolve issues with 2D decomposed UPP interface in regional UFS regression tests.
  4. Kate has a PR to document George's 2D decomposition documentation using doxygen
  5. George will review Kate's PR
  6. Huiya will review George's documentation
HuiyaChuang-NOAA commented 2 years ago

@JesseMeng-NOAA @BoCui-NOAA @WenMeng-NOAA @GeorgeVandenberghe-NOAA @fossell Summary from 4/11/2022 tag-up:

  1. Jesse fixed a bug in the regional UPP inline interface which contributed to issues with his earlier regional UFS RTs.
  2. Jesse would like to have someone else conduct RTs using 2D decomposition branch to confirm all RTs are passed
  3. Bo is working on merging all latest UPP develop changes to 2D decomposition branch. She's a bit concerned about just accepting "auto merge". She's working on resolving conflicts.
  4. Wen thinks "auto merge" mentioned by Bo above should be safe. She suggested starting with her latest unified FV3 interface and apply 2D decomposition on this subroutine; instead of trying to resolve all conflicts.
  5. Wen is working on an unified inline UPP interface.
  6. Huiya spoke to NCAR GTG group about running GTG inline but the consequence it more people will be able to access GTG code. NCAR didn't like the ideas so GTG will still be run in stand-alone mode in GFS V17.

Action items

  1. Huiya will ask GFS, RRFS, and HALF implementation managers to test 2D decomposition branch per Wen's suggestions
  2. Jesse will send Huiya instructions on how to test 2D decomposition branch when running UPP inline
  3. George will review Huiya's comments
  4. Kate will provide George with doxygen output
  5. Huiya will email Rahul that GTG still needs to be generated in stand-alone post in GFS V17.