This issue records results of testing ncrcat and cdo merge to concatenate NetCDF files. The NC files contain ~23 variables, covering one calendar year each. Each tool was given 2003 and 2004 data files to concatenate, noting the key indicators of runtime and output file size.
Results from different tools are:
Bare ncrcat : took 25m42s, resulting file was 710mb.
Bare cdo merge: stopped after a few minutes as the output was ~20GB.
cdo merge, zip level 7: took 11m74s, resulting file was 1340mb.
cdo merge, zip level 9: took 39m10s, resulting file was 1270mb.
Comments
ncrcat indicates the best compression overall, but slower than cdo merge/zip 7
CDO on my system may be compromised due to some older dependencies
The performance & compression results indicate something is happening under the hood. Is ncrcat written more efficiently and using better compression algorithms?
This issue records results of testing
ncrcatandcdo mergeto concatenate NetCDF files. The NC files contain ~23 variables, covering one calendar year each. Each tool was given 2003 and 2004 data files to concatenate, noting the key indicators ofruntimeandoutput file size.Results from different tools are:
ncrcat: took25m42s, resulting file was710mb.cdo merge: stopped after a few minutes as the output was~20GB.cdo merge, zip level 7: took11m74s, resulting file was1340mb.cdo merge, zip level 9: took39m10s, resulting file was1270mb.Comments
ncrcatindicates the best compression overall, but slower thancdo merge/zip 7CDOon my system may be compromised due to some older dependenciesncrcatwritten more efficiently and using better compression algorithms?