This issue records results of testing ncrcat and cdo merge to concatenate NetCDF files. The NC files contain ~23 variables, covering one calendar year each. Each tool was given 2003 and 2004 data files to concatenate, noting the key indicators of runtime and output file size.
Results from different tools are:
Bare ncrcat : took 25m42s, resulting file was 710mb.
Bare cdo merge: stopped after a few minutes as the output was ~20GB.
cdo merge, zip level 7: took 11m74s, resulting file was 1340mb.
cdo merge, zip level 9: took 39m10s, resulting file was 1270mb.
Comments
ncrcat indicates the best compression overall, but slower than cdo merge/zip 7
CDO on my system may be compromised due to some older dependencies
The performance & compression results indicate something is happening under the hood. Is ncrcat written more efficiently and using better compression algorithms?
This issue records results of testing
ncrcat
andcdo merge
to concatenate NetCDF files. The NC files contain ~23 variables, covering one calendar year each. Each tool was given 2003 and 2004 data files to concatenate, noting the key indicators ofruntime
andoutput file size
.Results from different tools are:
ncrcat
: took25m42s
, resulting file was710mb
.cdo merge
: stopped after a few minutes as the output was~20GB
.cdo merge, zip level 7
: took11m74s
, resulting file was1340mb
.cdo merge, zip level 9
: took39m10s
, resulting file was1270mb
.Comments
ncrcat
indicates the best compression overall, but slower thancdo merge/zip 7
CDO
on my system may be compromised due to some older dependenciesncrcat
written more efficiently and using better compression algorithms?