does spack install bufr_util and bufr_dump?

[edwardhartnett]

Are these utilities of the library?

Does the spack install include them?

This question comes from the NCO/EMC effort to use spack-stack on WCOSS2...

[jbathegit]

Are these utilities of the library?

The answer is half yes and half no.

Half yes because the latest version of the bufr_util package only contains one utility "apxdx", and that utility is now included with the NCEPLIBS-bufr library distribution as of the latest v12.1.0 (see #585).

Half no because the bufr_dump package doesn't have any similar connection to the library, other than pointing to some applications which do link to the library during compilation, but which otherwise aren't direct utilities of the library.

Does the spack install include them?

No, the spack install for NCEPLIBS-bufr doesn't currently include either of these packages, and I really don't see any reason for that to change.

Going forward, the only parts of bufr_util that still need to be maintained are the run_apxdx.sh and apdx.sh scripts which are specific to WCOSS2, and which therefore are probably not good candidates to be part of spack-stack. Similarly, the applications in bufr_dump are also very WCOSS2-specific and wouldn't be applicable to any other platform unless that filesystem also had copies of all of our observational data tanks and other supporting infrastructure.

[edwardhartnett]

OK, sorry, I thought these were part of NCEPLIBS-bufr, but they are actually different packages? Where are the repos?

We need to build these with spack so that NCO can use spack to install them on WCOSS2. So if you could point out the repos we can start by getting some spack package files together...

[jbathegit]

bufr_dump is available at https://github.com/NOAA-EMC/bufr-dump

There isn't a separate repo for bufr_util, because as noted earlier it only contains two scripts which are used to run the apxdx utility, and the utility itself is now part of the library distribution. If you want copies of the latest versions of those two scripts, you can grab them from /apps/ops/prod/nco/intel/19.1.3.304/bufr_util.v1.1.0/ush on WCOSS2.

[edwardhartnett]

Who is in charge of bufr-dump?

There is no code manager listed in the README...

[jbathegit]

Who is in charge of bufr-dump?

There is no code manager listed in the README...

I believe it's @ilianagenkova. She's the contractor lead for the EMC obsproc team. But she's also on leave for the rest of this week, so if I'm wrong then it will likely be at least next Monday before she can chime in to correct me :-)

[ilianagenkova]

I am indeed the manager of bufr-dump. What is spac and do we/obsproc really need it (bufr-dump is one of a few obsproc-related repos - some are considered packages and others are modules)

[edwardhartnett]

Hello @ilianagenkova !

Spack is a package management tool which is used to install the UFS and many other HPC systems.

Spack is only important if you are writing code that runs operationally on WCOSS2. Does your code do that?

[ilianagenkova]

Our code is one of the many in $PACKAGEROOT that all run on wcoss2.

[edwardhartnett]

In that case, we need a spack package file. We will submit a PR...

[ilianagenkova]

Is our code singled out for some reason, or every code in $PACKAGEROOT at some point would need to be spack-compliant. Time is a commodity that we don't have. As Jeff said, there is likely no reason for us to start using spack.

I am not against doing things right, I'm just highlighting priorities and timelines, and lack of any experience with spack (to be very honest). Would someone from NCEPLIB-bufr or EIB team help us with spack?

[edwardhartnett]

I believe we will need a spack package file for everything that's installed on WCOSS2. (I will confirm that.)

We (NCEPLIBS team) will do the spack package files. We will help you maintain them and give you some training if you want to know more. Ideally, the spack package file will be part of your repo, and be testing by a CI system, but I see that you don't see to have a CI system at this time.

WCOSS2 is moving to using spack for all installs. This is part of an NCO-EMC effort over the next few months to make sure that everything can be installed cleanly with spack. Moving to spack will allow NCO, GDIT, and EMC to handle installs uniformly and correctly every time. We'll provide all the help needed to make the transition.

[jbathegit]

I believe we will need a spack package file for everything that's installed on WCOSS2. (I will confirm that.)

Ed, could you please confirm that? This is the first I've heard that EVERYTHING(!?) installed on WCOSS2 now needs to use Spack(!?) My understanding was that Spack was only important for libraries and other applications that run on a number of external platforms besides just WCOSS2, not for application codes that only ever run on WCOSS2. I wasn't even aware that correct builds and installs were considered by NCO to be a widespread problem for application codes on WCOSS2. So unless I'm wrong about that, then this seems like a solution in search of a problem, and therefore a lot of overkill and unnecessary work.

[edwardhartnett]

We are in talks with NCO now for switching to spack. They have expressed an interest in using spack for everything. This package (bufr_util/bufr_dump) was listed on the spreadsheet of those packages missing a spack build (and it's a short list, almost everything can be built by spack).

I will keep you up to date on discussions with NCO. We are meeting with them next week to go over the list of packages missing from spack. I will discuss bufr_util/bufr_dump with them.

[ilianagenkova]

FWIW, my team is working on updating bufr_dump with tentative release to NCO in October. It will be released as one of 3 repos that work together (all 3 will be updated). I don't know if that time will be enough for adding spack now.