Repository for the development of ADRIA: Adaptive Dynamic Reef Intervention Algorithms. ADRIA is a multi-criteria decision support tool set particularly useful for informing reef restoration and adaptation interventions.
1
stars
0
forks
source link
Modify scatter-gather approach to support large(r) datasets #48
I naively assumed that data written to disk could be read back in with a smaller, more manageable, memory footprint.
Unfortunately this does not seem to be the case as MATLAB crashes with error when collating results:
MatlabExecutionError:
File D:\[...]\ADRIA_repo\ADRIAfunctions\IOHandler\collectDistributedResults.m, line 63, in collectDistributedResults
Requested 25x36x26x70240x20 (244.9GB) array exceeds maximum array size preference (63.9GB). This might cause MATLAB to become unresponsive.
Either we pass in a function handle and calculate metrics as data is read in (at the cost of flexibility) or we implement a datastore.
ConnectedSystems
commented
2 years ago
I naively assumed that data written to disk could be read back in with a smaller, more manageable, memory footprint.
Unfortunately this does not seem to be the case as MATLAB crashes with error when collating results:
Either we pass in a function handle and calculate metrics as data is read in (at the cost of flexibility) or we implement a datastore.
Seems the former is the easiest option.