search

aiddata / geo-datasets

Scripts for preparing datasets in GeoQuery
http://geoquery.org
MIT License
20 stars 11 forks source link

Explore rasterio COG generation memory usage #158

Open sgoodm opened 1 year ago

sgoodm commented 1 year ago

COG generation seems to use significantly more memory than a traditional GeoTIFF when using Rasterio. I suspect this is either tied to the overlay creation, or the fact that we are often restructuring the raster blocks and have to read / write larger chunks than the final COG blocks (usually 512x512)

jacobwhall commented 2 months ago

I noticed significant memory usage (8Gb+) when processing a single VIIRS raster this morning. This will limit the number of concurrent COG exports we can run, even on the HPC.

jacobwhall commented 2 months ago

Just ran a more formal test and observed a VIIRS average mask COG exports using 12+ gigabytes of RAM. Similar result with and without LZW compression explicitly enabled in the **meta kwargs.