search

open-AIMS / ADRIA_matlab

Repository for the development of ADRIA: Adaptive Dynamic Reef Intervention Algorithms. ADRIA is a multi-criteria decision support tool set particularly useful for informing reef restoration and adaptation interventions.
1 stars 0 forks source link

Write results to file as they complete (reduce memory use) #106

Closed ConnectedSystems closed 2 years ago

ConnectedSystems commented 2 years ago

As in title, changed the scatter/gather approach to write results to file as they complete as opposed to in batches.

This new approach may result in runs being slower than before as writing results can take time and this happens more frequently now, but saves memory.

Batch result netCDFs now store data in groups, and the gather function has been modified to support both the new and old structure.

One minor change that will require additional changes elsewhere is that the MCDA_rankings field name has been renamed to site_rankings so that all variable names, string IDs and field names are all consistent with one another.

EDIT:

A related set of changes already merged into main includes vectorization of the growthODE_KA() function. This brings down the time per run to about 0.7 - 0.8 seconds compared to ~1.3 - 1.5 seconds on my laptop (when running batches).

Rosejoycrocker commented 2 years ago

If anything has changed since you submitted your request for review let me know, otherwise I'll merge your pull request.

ConnectedSystems commented 2 years ago

Thanks Rose, there will be more changes incoming as I'm still tweaking things but I will put those in a separate PR.

To make sure your attention is drawn to this note:

One minor change that will require additional changes elsewhere is that the MCDA_rankings field name has been renamed to site_rankings so that all variable names, string IDs and field names are all consistent with one another.

This will affect your other scripts: when collecting logs, you'll have to change MCDA_rankings to site_rankings

Rosejoycrocker commented 2 years ago

Thanks, I'll finalise the pull request. Thanks for bringing my attention to the name change, I'll make sure this has changed in my scripts.