search

NOAA-GFDL / FRE-NCtools

Tools for manipulating and creating netCDF inputs for FMS managed models
GNU Lesser General Public License v3.0
20 stars 28 forks source link

Let split_ncvars split large Netcdf-4 files #312

Closed ceblanton closed 1 month ago

ceblanton commented 2 months ago

resolves #311

ilaflott commented 1 month ago

The default behavior was to take a netCDF file input, and create 64-bit offset netCDF3. Perhaps that is not desired in the case of issue #311, but do we know where this is desired? do we know where people are expecting this tool to output specifically a netCDF3 file?

more to the point: why is the proposed solution to change the default behavior, rather than adding an extra flag to achieve the desired behavior?

ilaflott commented 1 month ago

Another thought- given that the desired behavior is input_file_type == output_file_type, if the input file is of the 64-bit offset variety enabling larger file output, then isn't removal of the ncks version checking logic re-introducing potential for failures when the version of ncks isn't quite what's expected? wouldn't we need that logic to preserve the offset?

ceblanton commented 1 month ago

Thanks Ian. Good questions. Your last question is the easiest. The NCO version checking is only needed for outputting the 64-bit NC3 format, so if we remove that option, the NCO version can be ignored. (That's a nice document about netcdf, there!)

I see your point about flexibility. If split-ncvars.pl were used for multiple purposes and some of the purposes were format dependent, then I would agree with an --format option. But FRE is the only known user of split-ncvars, and if there is an outcry we can add an option (But.. then we'd have to check the NCO version probably). Users can always use nccopy or even ncks to convert formats.

The main reason to not add an extra flag to get the desired format behavior is that FRE would then have to check the version of the input file, just to pass it to the split tool. That wouldn't be terrible but would be unnecessary, since the desired behavior should be to output the same format as the input. That is the convention used by other tools, such as mppnccombine, fregrid, and timavg.csh.