TUDelftGeodesy / stmtools

Xarray extension for Space-Time Matrix
https://tudelftgeodesy.github.io/stmtools/
Apache License 2.0
6 stars 0 forks source link

memoryError on dask.Dataset larger than ~500,000,000 points (32GB memory) #32

Open Simon-van-Diepen opened 1 year ago

Simon-van-Diepen commented 1 year ago

When creating a Dask Dataset as space-time matrix, one index (int64) for each entry is always computed. On a 32GB machine this will cause a memory overflow for lazy, not-computed stms larger than ~500,000,000 points, yielding the error Stream closed.