upp, g2, potentially other libraries out of date in new Cheyenne location for HPC-stack

[mkavulich]

Please describe the package or library you would like to add to hpc-stack. With the resolution of #407, the hpc-stack build on Cheyenne was moved to a new location (/glade/work/epicufsrt/GMTB/tools/hpc-stack-v1.2.0_6eb6/modulefiles). However, while this build is more up-to-date than the old location (/glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.2.0/modulefiles) in some libraries (e.g., esmf/v8.3.0b09 is available in the new location and not the old), for others the new location is out of date.

For example:

module av g2

-------------------------- /glade/work/epicufsrt/GMTB/tools/hpc-stack-v1.2.0_6eb6/modulefiles/compiler/intel/2021.2 ---------------------------
   g2/3.4.2 (D)    g2c/1.6.2    g2tmpl/1.10.0 (D)

------------------------------- /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.2.0/modulefiles/compiler/intel/2021.2 -------------------------------
   g2/3.4.2    g2/3.4.3    g2c/1.6.4 (D)    g2tmpl/1.10.0

The latest UFS_UTILS (and therefore, UFS SRW App) requires g2/3.4.3, which is not available in the /glade/work/epicufsrt/ space. As an additional example, upp/10.0.8 is the latest version in this /glade/work/epicufsrt/ space, but the UFS SRW App is using upp/10.0.10

There may be other out-of-date library versions that I have missed.

Additional context

More of a question, but why has the location been moved in the latest update? Should we be pointing to this newer location for all our NCEPLIBS needs? How is this being communicated?

[jkbk2004]

@Hang-Lei-NOAA @mkavulich I am looking into this issue. I will keep you posted.

[WenMeng-NOAA]

The UPP is configurated as a submodule of FV3 component in ufs-weather-model for supporting in-line post. I would think no upp lib installation needed.

[christopherwharrop-noaa]

@jkbk2004 / @Hang-Lei-NOAA The w3emc package also needs updating to include both w3emc-2.9.1 and w3emc-2.9.2 in order to support newer versions of GSI and RRFS SRW which includes the GSI.

[christopherwharrop-noaa]

@WenMeng-NOAA - If UPP is going to be a hpc-stack package, then I think it needs to be updated appropriately. There could easily be other software that need it outside the context of the FV3 component.

[WenMeng-NOAA]

@christopherwharrop-noaa The UPP is now configurated as a submodule of FV3 component in the ufs-weather-model for in-line post processing.

[christopherwharrop-noaa]

@WenMeng-NOAA - Yes, you mentioned that. My question is what about applications/software outside the scope of ufs-srweather-model that need UPP? Where do they get it? If no such usage exists anywhere in the entire UFS ecosystem then maybe UPP doesn't need to be a hpc-stack package. But unless that is the case, hpc-stack still needs to maintain versions of UPP in its various installations.

[WenMeng-NOAA]

@christopherwharrop-noaa The applications outside of the ufs-weather-model scope can directly checkout the UPP and run standalone post processing (off-line post). The upp lib was used for supporting in-line post in the ufs-weather-model. Now we changed the configuration of in-line post in ufs-weather-model.

[christopherwharrop-noaa]

@WenMeng-NOAA - Right. Ok. The UPP is also a components of the UFS Apps, which use manage_externals to get it. So, perhaps support for UPP should be removed from hpc-stack? I'm not sure I know enough about the overall UFS ecosystem to say whether that would be a good idea. However, it stands to reason that if UPP is part of hpc-stack, then hpc-stack should maintain its installation.

[kgerheiser]

Once UFS switched to using UPP internally we stopped installing it. If there's still a need for it externally we can still support that, and if it's not maybe it should be removed.

[christopherwharrop-noaa]

@kgerheiser - I see what @WenMeng-NOAA was talking about now. Sorry for my misunderstanding earlier. At this point you've convinced me it isn't needed.

[christopherwharrop-noaa]

@kgerheiser - Just to clarify, though, g2 and w3emc (2.9.1 and 2.9.2 please) still need to be installed in the new hpc-stack location. I wanted to re-up that since conversation got away from that here.

[kgerheiser]

@jkbk2004 handles the Cheyenne build. I don't have access there.

[jkbk2004]

@kgerheiser @christopherwharrop-noaa I am going to create new PRs on srw to move hpc-stack to epicufsrt so that we can follow up quickly. I am combining hpc-stack sync case between srw and ufs-wm as well. RT test is on-going to confirm.

[christopherwharrop-noaa]

@jkbk2004 - Just to let you know, the current SRW does not use newer w3emc, but we do need those newer ones on Cheyenne for upcoming changes to SRW for RRFS.

[jkbk2004]

@christopherwharrop-noaa ufs-wm is based on hpc-stack at /glade/work/epicufsrt/GMTB/tools/hpc-stack-v1.2.0_6eb6 but I am mtrying to move to /glade/work/epicufsrt/GMTB/tools/gnu and intel to cover the new requirement. I think I will create srw PRs tomorrow or at least over weekend.

[jkbk2004]

@christopherwharrop-noaa on ufs-wm side, I like to update jasper/g2/libpng across all platforms. The PR is likely next week. And then srw side, the rest of the story is about fms/esmf and new hpc-stack location and requirements.

[christopherwharrop-noaa]

@jkbk2004 - Ok. But, this issue is about consistency between the old hpc-stack install and the new one. What @mkavulich wanted fixed was to make sure that the new location contains the same packages and versions as the old one. He reported g2 and UPP discrepancies. I reported w3emc discrepancy. Can you please tell us what you're planning to do to to make the new location contain the same things as the old one?

[jkbk2004]

@christopherwharrop-noaa I agree. g2 and upp at new location should stay same as old location. but I am focusing on fms/esmf. srw env uses fms2021.03 and esmf8.2.0 I think upgrading to fms/2022.01 and esmf8.3.0b09 should have impact on srw side. But I will try to confirm. I don't see any issue with other packages such as w3emc/ncio to add newer version to new location.

[christopherwharrop-noaa]

@jkbk2004 - Ok. Thank you. Hopefully the g2 and w3emc updates can be done quickly as RRFS and GSI will not build on Cheyenne without those.

[kgerheiser]

@jkbk2004 has done a lot of work on Cheyenne. Is this complete?