explain DA and 4/8/d builds in README

[edwardhartnett]

@jbathegit can you tell me about the DA build and then I will explain it in the README?

Sorry if you have explained it to me before - if so I failed to capture it in the documentation and need to add it.

Some builds use dynamic memory and some don't? I see this in the CMakeLIsts.txt:

# Determine the kinds upfront (both src/ and test/ need them).
# Both GNU and Intel support DYNAMIC_ALLOCATION
list(APPEND kinds "4_DA" "8_DA" "d_DA")
# Only Intel supports STATIC_ALLOCATION
if(CMAKE_C_COMPILER_ID MATCHES "^(Intel)$")
  list(APPEND kinds "4" "8" "d")
endif()

Then in the src/CMakeLists.txt we have:

  # determine ALLOCATION based on kind
  if(${kind} MATCHES "^([4|8|d]_DA)$")
    set(allocation_def "DYNAMIC_ALLOCATION")
  else()
    set(allocation_def "STATIC_ALLOCATION")
  endif()

So we are building some versions of the library with and without DA?

[edwardhartnett]

While at it, let's capture the details of the 4, 8, and d builds as well. I only understand it vaguely.

Are the 4 builds with 4-byte reals, the 8 with 8-byte? Then what's the d?

Also does this also affect integers? These are all achieved with fortran flags, right?

[jbathegit]

Back many years ago when the BUFRLIB was first developed, there were only static allocation builds, which means that all internal BUFRLIB array space is allocated at compile time. But we had many users who were working with different integer and real sizes in their application programs, so we supplied various BUFRLIB compilations to meet their needs. Those still persist to this day, and yes these are all controlled with Fortran and C compiler flags:

• "4" = 4-byte integers and 4-byte reals
• "8" = 8-byte integers and 8-byte reals
• "d" = 4-byte integers and 8-byte reals (sort of a hybrid)

This worked fine for a while, but then as BUFR become more widespread and entrenched, we started to have users who needed larger arrays to work with certain voluminous data types. For a while, we provided an additional "s" build (short for "supersize") which had much larger allocated arrays, but this was an imperfect long-term solution because there would invariably be some user who needed one array a bit larger, or another who needed a different array slightly larger, and we could never seem to accommodate everyone with a single, static supersize build given how rapidly the use of BUFR was expanding for so many varied types of observational data. So about 7 years ago I went back to the proverbial drawing board and basically dove down the proverbial rabbit hole for several months, and I ended up refactoring the entire library to allow for dynamic allocation of BUFRLIB arrays.

What this meant is that users could now link the "DA" version of any of the above compilations, which in turn allowed them to call a new subroutine isetprm() to pick whatever array sizes they wanted or needed for any number of various internal BUFRLIB arrays, because these arrays would now be allocated at run time rather than at compile time. This also allowed us to eventually retire the "s" (supersize) build, since that was just the same thing as a dynamic allocation build with certain array sizes preset. But we still kept the original static allocation builds for users who didn't need any special large array sizes, since that way they could continue to use those and not have to pay the additional cost of dynamic array allocation at run time every time they ran their program. In other words, dynamic allocation builds were now available for those who really needed this flexibility above and beyond the default internal array sizes, but those who were fine with the latter could still use the more efficient static allocation builds.

[edwardhartnett]

OK, thanks for that clear explanation, which I will transfer to the readme.

We will be doing away with the 4/8/d system in all NCEPLIBS. It's not clear when that change will be proposed for this repo - we're going to try it out in the simpler sp repo first.

It seems unlikely that dynamic allocation is measurably slower. I would like to see proof of that before continuing to support the static allocation builds moving forward. Seems like the DA build should become what everyone uses. This is similar to other libraries on the I/O stack which face these choices.

[jbathegit]

Hmm, OK that's news to me. If there's any sort of plan to do away with the 4/8/d system and/or static builds for all NCEPLIBS(?), then that's a decision that would also need to be vetted and cleared across our WCOSS and RDHPCS user communities, at least insofar as the BUFRLIB is concerned. That's a huge community of users with a long-term history and vested interest in the library, so a huge corresponding number of operational codes would need to transition if we were to stop supporting certain builds.

[edwardhartnett]

We will discuss. We believe we have a solution that will have no code impact (i.e. be fully backward compatible). We're going to try it out.