Unidata / netcdf-c

Official GitHub repository for netCDF-C libraries and utilities.
BSD 3-Clause "New" or "Revised" License
520 stars 262 forks source link

classic only build not dealing with filters/quantization well #3020

Open edwardhartnett opened 2 months ago

edwardhartnett commented 2 months ago

For a classic only build there should be no filters or quantization.

# General
-------
NetCDF Version:     4.9.4-development
Dispatch Version:       5
Configured On:      Thu Sep 12 06:47:50 MDT 2024
Host System:        x86_64-pc-linux-gnu
Build Directory:    /home/ed/netcdf-c
Install Prefix:         /usr/local/netcdf-c-classic
Plugin Install Prefix:  /usr/local/hdf5/lib/plugin

# Compiling Options
-----------------
C Compiler:     /usr/bin/gcc
CFLAGS:          -fno-strict-aliasing
CPPFLAGS:       
LDFLAGS:        
AM_CFLAGS:      
AM_CPPFLAGS:        
AM_LDFLAGS:     
Shared Library:     yes
Static Library:     yes
Extra libraries:    -lm -lz -lsz -lzstd -lxml2 -lcurl 
XML Parser:             libxml2

# Features
--------
Benchmarks:     no
NetCDF-2 API:       yes
HDF4 Support:       no
HDF5 Support:       no
CDF5 Support:       yes
NC-4 Parallel Support:  no
PnetCDF Support:    no

DAP2 Support:       no
DAP4 Support:       no
Byte-Range Support: no

S3 Support:         no
S3 SDK:             none

NCZarr Support:     yes
NCZarr Zip Support:     no

Diskless Support:   yes
MMap Support:       no
ERANGE Fill Support:    no
Relaxed Boundary Check: yes

Plugins Enabled:        yes
Plugin Install Dir:     /usr/local/hdf5/lib/plugin

Quantization:       yes
Logging:            no
SZIP Write Support:     yes
Standard Filters:       bz2 deflate szip zstd
ZSTD Support:           yes
Parallel Filters:       no
edwardhartnett commented 2 months ago

I will submit a fix shortly...

czender commented 2 months ago

Whether or not there should be quantization is debatable. It's perfectly well defined and supportable by classic format, it's just that there is not any known use. I think we discussed whether classic should support quantization before, and I remember concurring that it is OK to turn it off for classic. My views have evolved since then. As long as it's working for classic, no reason to turn it off in my mind. YMMV. In any case, NCO supports quantization for classic via it's internal algorithms.

edwardhartnett commented 2 months ago

It does not work for classic.

Putting it in for classic would require diving into the libsrc code, which is not something I would choose to do without great need.

Of course it makes sense for classic, and would even provide benefit: a quantized classic file would gzip to a much smaller size.

My only concern here is that currently netcdf-fortran fails for classic-only netcdf-c library because quantize constants are missing. Obviously we want a classic only build to work.

So my thought was to detect classic only builds, and then to set quantization to "no" (since there is no quantize for classic only). Then netcdf-fortran could see that setting and know not to include the quantize constants.

czender commented 2 months ago

OK, nevermind me, I misinterpreted your original message. Sounds like we're in agreement on the hypothetical value of having quantization in classic files.