hunzikp / velox

https://hunzikp.github.io/velox/
119 stars 23 forks source link

Can not allocate vector of size N when working with large rasters. #10

Closed AronBoettcher closed 6 years ago

AronBoettcher commented 6 years ago

I'm trying to speed up a summarize by polygon function using your package, but I work with extremely large rasters, generally 2-3gb geoTiffs. When I try to convert a raster stack object to a velox object, I get ye' olde warning "Can not allocate vector of size N when working with large rasters.", I assume because velox is trying to place the entire uncompressed tif into memory (huge). Its way beyond my skillset, but it would be amazing to be able to somehow use the velox functions on very large rasters.

hunzikp commented 6 years ago

Hi Aron, thx for the comment.

Yes, the error arises because velox attempts to load all data in memory. In fact, this is one of the reasons why velox is faster (by default) than the raster package, which implements a very smart strategy for dealing with large files. I'm not currently planning to replicate this functionality, so unfortunately I can only suggest you either use raster, attempt to optimize memory usage (do you need all files loaded at once?), or work on a more powerful machine. Cheers, ph