search

NOAA-EMC / hpc-stack

Create a software stack for HPC's
GNU Lesser General Public License v2.1
30 stars 36 forks source link

Add bufr_util to hpc-stack #176

Closed ShelleyMelchior-NOAA closed 3 years ago

ShelleyMelchior-NOAA commented 3 years ago

In an effort to get all obsproc and decoder related utilites encapsulated in hpc-stack, please add bufr_util.

$ module show bufr_util/1.0.1
------------------------------------------------------------------------------------------------------------------------------
   /gpfs/dell1/nco/ops/nwprod/modulefiles/compiler_prod/ips/18.0.1/bufr_util/1.0.1:
------------------------------------------------------------------------------------------------------------------------------
whatis("This module sets the environment variables for BUFR utilities.  ")
prereq_any("ips")
prereq_any("prod_util")
setenv("BUFR_UTIL_VER","v1.0.1")
setenv("USHbufr_util","/gpfs/dell1/nco/ops/nwprod/bufr_util.v1.0.1/ush")
prepend_path("PATH","/gpfs/dell1/nco/ops/nwprod/bufr_util.v1.0.1/ush")
setenv("RUN_APXDX","/gpfs/dell1/nco/ops/nwprod/bufr_util.v1.0.1/ush/run_apxdx.sh")
setenv("EXECbufr_util","/gpfs/dell1/nco/ops/nwprod/bufr_util.v1.0.1/exec")
prepend_path("PATH","/gpfs/dell1/nco/ops/nwprod/bufr_util.v1.0.1/exec")
setenv("DEBUFR","/gpfs/dell1/nco/ops/nwprod/bufr_util.v1.0.1/exec/debufr")
setenv("XBFMG","/gpfs/dell1/nco/ops/nwprod/bufr_util.v1.0.1/exec/xbfmg")
help([[Sets environment veriables for BUFR utilities.

]])
aerorahul commented 3 years ago

@ShelleyMelchior-NOAA The DEBUFR is a utility already built in the bufr library. Perhaps we should work to add XBFMG to the library utils, if it is generic and usable by the community.

These paths are specific to a particular version, and for a particular machine.

This package (hpc-stack) is supposed to be generic and the components of this package are expected to be usable on any machine.

ShelleyMelchior-NOAA commented 3 years ago

DEBUFR and XBFMG are not the only entities of BUFR_UTIL. What is more, BUFR_UTIL will be expanding once we move to WCOSS2. That is where dumpjb and its associated utilities will be residing.

Much of obsproc and decoders is platform specific, i.e., operational supercomputers. The software will not run on non-operational machines b/c of the need for operational tanks and/or networking infrastructure to bring in the data. If that makes these utilities unsuitable for hpc-stack, then that's OK. We were just trying to conform to the new paradigm.

Perhaps before moving forward we need a broader conversation w/ both the decoder team and NCO. I admit that hpc-stack is new to me. I am only just now grasping its purpose and may have some things misconstrued.

edwardhartnett commented 3 years ago

hpc-stack is new and we are working though exactly how it should be used. ;-)

If this code is needed operationally, and is sufficiently general that it will be used in multiple places, then it might be suitable for some form of inclusion in hpc-stack.

However, most likely that would not be as a separate package, but as part of the already existing NCEPLIBS-bufr. In that case, we would need the code to be properly documented and we need some test code.

arunchawla-NOAA commented 3 years ago

ufs-utils are utility codes that use hpc-stack modules to build the executables that are then used by the workflow. If bufr-utils are going to be a set of executables then they should probably be built outside hpc-stack. If however you want these to be part of a library package then it makes sense to be part of hpc-stack.

ShelleyMelchior-NOAA commented 3 years ago

Ok, I touched base w/ Steven Earle to get the NCO take on this. Provided is his email response to my query. I have highlighted his main point, which makes sense to me and its application to obsproc.

Is hpc-stack going to be adopted in operations on WCOSS2?

Yes, parts of hpc-stack will be running in operations on WCOSS2.

In our brainstorming for a condensed obsproc network on WCOSS2, we talked about moving dumpjb and its supportive components into bufr_util. Presently bufr_util is not integrated into hpc-stack. Should it be? Or should it remain standalone, like prod_envir, since it is mostly restricted to operational machine use (b/c of dependence upon BUFR tanks)?

It doesn't really make sense to me to have dumpjb as part of hpc-stack, but some other bufr utilities maybe do make sense... like a debufr. Utilities like dumpjb that are wcoss specific (pulling data out of tanks that only exist on wcoss) then I don't think hpc-stack is the home for them. That said, if it just comes along with a bigger bufr-util package then I don't have a problem with it being in there. I would like to have prod-util part of hpc-stack, but not prod-envir; for the same reasons as bufr utils. If they are wcoss specific then stand alone, if they are valuable on development/community systems then hpc-stack.

The take-away is that bufr_util should not be added to hpc-stack. Some members of bufr_util, like debufr, have a home in hpc-stack, via NCEPLIBS-bufr, and that is suitable per the standard rule in bold in Steven's response.

arunchawla-NOAA commented 3 years ago

Thank you Shelley!

This is great input, and whole heartedly agree. So we should probably add an issue in NCEPLIBS to add the relevant bits to NCEPLIBS-bufr

jbathegit commented 3 years ago

Thanks @ShelleyMelchior-NOAA for bringing me in on this thread. My $0.02 is that debufr makes sense to have as a utility in NCEPLIBS-bufr, and as noted above it's already included in the package. The only other utility that might(?) have some widespread usefulness is xbfmg, which we can maybe think about adding to NCEPLIBS-bufr in the future. But I don't think apxdx or dumpjb are suitable utilities for use outside of WCOSS and therefore shouldn't be part of any distribution package to the wider community.

aerorahul commented 3 years ago

@ShelleyMelchior-NOAA Thank you for following up with WCOSS.

kgerheiser commented 3 years ago

I'm going to close this issue given the response here: https://github.com/NOAA-EMC/hpc-stack/issues/176#issuecomment-788977880