Add extdata tests to test compression, bit-shaving, and quantization

[mathomp4]

Types of change(s)

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)
• [ ] Trivial change (affects only documentation or cleanup)
• [ ] Refactor (no functional changes, no api changes)

Checklist

• [x] Tested this change with a run of GEOSgcm
• [x] Ran the Unit Tests (make tests)

Description

Recently, @edwardhartnett asked if MAPL had some sort of unit test that tested our compressed output. And, nope, we did not have any.

So, this PR adds 5 new ExtData test cases to allow us to test writing of History with compression, bit-shaving, and quantization. The new tests are all "copies" of case1 with additions to the HISTORY1.rc file in each:

30. Case1 with deflate compression
31. Case1 with deflate compression and MAPL bit-shaving
32. Case1 with deflate compression and NetCDF bitgroom quantization (only enabled if netcdf built with quantization support)
33. Case1 with deflate compression and NetCDF bitround quantization (only enabled if netcdf built with quantization support)
34. Case1 with deflate compression and NetCDF granular_bitround quantization (only enabled if netcdf built with quantization support)

As the list says, the last three are only turned on if we detect NetCDF quantize support was built.

Related Issue

[mathomp4]

ok. As long as we are making changes here, it would be very useful if all the cases used 2 digits. It has been problematic in the past run "case2" because it maches "case2*".

Ahh. I can look at this. Let me do some fun with f2 and sed...

[edwardhartnett]

OK, but how about zstandard compression?

[mathomp4]

OK, but how about zstandard compression?

@edwardhartnett We can't add a test for it because we have no support for it in MAPL. Period.

We can work toward it, but that will take a while because we'd need to make sure we can build it correctly in our libraries, etc.

If you can help me with all the steps needed for that, I can work on it. I know I know there are a lot of changes needed to support it, right? I think have to manually set --enable-zstandard-plugin for autotools and netcdf-fortran.

But perhaps I need to convert to CMake? When I look at netcdf-fortran CMake, it seems to default to enabling zstd support:

OPTION(DISABLE_ZSTANDARD_PLUGIN "Disable ZStandard plugin detection and functionality, even if libnetcdf was compiled with plugin support" OFF)

But before that I think I need start building hdf5_plugins as well? And as far as I can see, you can only get the zstandard plugin via CMake since configure.ac doesn't have any mention of zstandard.

...And so that probably means I need to set up HDF5 to build with CMake maybe? It might be an effort.

[edwardhartnett]

Building is a separate topic, but you can check for the presence of zstd the same way (I presume) you check for quantization, the netcdf_meta.h file. There is a HAS_ZSTD macro there if zstd is supported.

To use it, from F90, just change the deflate_level parameter in nf90_def_var() to zstandard_level.

In C, call nc_def_var_zstandard() instead of nc_def_var_deflate().

[mathomp4]

Building is a separate topic, but you can check for the presence of zstd the same way (I presume) you check for quantization, the netcdf_meta.h file. There is a HAS_ZSTD macro there if zstd is supported.

To use it, from F90, just change the deflate_level parameter in nf90_def_var() to zstandard_level.

In C, call nc_def_var_zstandard() instead of nc_def_var_deflate().

@edwardhartnett Yes, I know it would be fairly simple to add support to MAPL, but I'm reluctant to add code to MAPL supporting an option I cannot yet test.

If there is a simple way of building netcdf-fortran using spack that has zstd support, then I could prototype under that.

[edwardhartnett]

Yes, spack builds netcdf-c and netcdf-fortran with zstd.