search

PCMDI / cmor

Climate Model Output Rewriter
BSD 3-Clause "New" or "Revised" License
53 stars 32 forks source link

Expose NetCDF quantize and zstandard functions #751

Closed mauzey1 closed 3 months ago

mauzey1 commented 3 months ago

Resolves #725

NetCDF variable quantization and Zstandard compression can now be enabled in CMOR using the functions cmor_set_quantize and cmor_set_zstandard.

Zstandard compression will only be applied to variables when deflate compression, which is used by default, is disabled. This is done to avoid errors caused by having two compression methods being applied.

durack1 commented 3 months ago

@piotr-florek-mohc is there any chance you could pull the 3.9 release to test across the MOHC demo cases?

durack1 commented 3 months ago

I presume that the test suite will be run before merging, right?

mauzey1 commented 3 months ago

@durack1 The nightly build will have the changes from this PR when it is merged into main.

I'm currently encountering an issue with the singleton test. It seems that adding the values for quantize and zstandard to the struct cmorvar are causing some kind of alignment error. I'll label this PR as a draft until it is fixed.

durack1 commented 3 months ago

@wachsylon any chance you could take the CMOR 3.9.0 nightly for a test drive and provide any feedback before we get this released - hopefully this PR will be merged soon, enabling a nightly with the new functionality to test drive

mauzey1 commented 3 months ago

@durack1 @wachsylon

I'm currently having issues with understanding why the macOS builds (x86 and arm64) are breaking on GitHub Actions. I was able to get macOS arm64 builds working in a CircleCI pipeline running macOS 14.3.1 with xcode 15.4.0 with this branch's changes. I was also able to get the builds working on my x86 Macbook running macOS 14.6.1 with xcode 15.4.0.

GitHub Action's macOS arm64 runner uses macOS 14.6.1 with xcode 15.4.0. I don't know why I'm getting errors since this configuration should be similar to the ones where I got working builds.

The specific error being that the values in the cmor_var_t struct seem to be misaligned by the addition of the variables zstandart_level, quantize_level, and quantize_nsd. https://github.com/PCMDI/cmor/blob/1e32b83c77c31342e35ac443c80e7ed9da52dc2f/include/cmor.h#L438-L501

I detected the errors in test_singletons.c where cmor_vars[var_id].ndims is set to -1, which is the value used in singleton_ids for unused IDs. If you tried looping through cmor_vars[var_id].singleton_ids using num_axes in place of cmor_vars[var_id].ndims in the for loop, you will see it going through -1 values until it reaches what appears to be the IDs in axes_ids. https://github.com/PCMDI/cmor/blob/1e32b83c77c31342e35ac443c80e7ed9da52dc2f/Test/test_singletons.c#L45-L54

I don't know what causes this issue nor why it only occurs in the GitHub Action builds and not anywhere else.

mauzey1 commented 3 months ago

I have found the problem. It turned out that the build in the PR was using a branch (main?) that was different from the source branch that contained the changes. This caused the misaligned values when building tests using the source branch's headers while using the PR build's library.

I have fixed it by making the PR build use the source branch.