Session fails depending on Chunk parameter

Nowosad / supercells

The goal of supercells is to utilize the concept of superpixels to a variety of spatial data.

https://jakubnowosad.com/supercells/

GNU General Public License v3.0

66 stars 5 forks source link

Session fails depending on Chunk parameter #35

Closed spono closed 6 months ago

spono commented 7 months ago

Supercells seems to behave differently according to the chunk size:

using chunk > 500 I get the following error: Error: I cannot return supercells. This may be due to a large number of missing values in the 'x' object. Try to either trim your data to the non-NA area (e.g., with 'terra::trim()') or increase the number of expected supercells.
while chunk <= 500 leads directly to the Rstudio session to crash.

This issue goes together with #34 .

Nowosad commented 7 months ago

Hi again @spono:

this problem is closely related to the previous one -- your superpixels are on average about 400 cells in size, so when you set the chunk size to be smaller than about 400 (I've tried chunk = 400 exactly and it worked) then it is impossible and the code (internal C++ code) fails.

In general, the chunk option is for large data (when the raster cannot be fit into RAM), so I would discourage using it for small data. Let me know if you have any suggestions of changes here.

Nowosad commented 6 months ago

I decided not to add any warnings/errors for this case -- using chunks is not aimed at small data (I hope people encountering it will be able to find this issue).