Initial allocated memory of 31GB per CPU would work fine with smaller netCDF files, however cog conversion fails if we input stackednetCDF files.
Also, insufficient memory also resulted in cogconversion errors while copying source overviews.
Proposed solution
1) Increase memory to 62GB per Node.
2) Enable copy_src_overviews to true as part of default profile template instead of enabling them within rasterio.shutil.copy process.
Request for this pull request
Initial allocated memory of
31GBperCPUwould work fine with smallernetCDFfiles, howevercogconversion fails if we inputstackednetCDFfiles. Also, insufficient memory also resulted incogconversionerrors while copying source overviews.Proposed solution
1) Increase memory to
62GBperNode. 2) Enablecopy_src_overviewsto true as part ofdefault profile templateinstead of enabling them withinrasterio.shutil.copyprocess.