search

aemon-j / LakeEnsemblR

An R package that facilitates multi-model ensembles for lake thermodynamics. Also includes tools for calibration, sensitivity analysis and data visualization.
GNU General Public License v2.0
32 stars 19 forks source link

Running into RAM trouble with deep lakes #156

Open tadhg-moore opened 4 years ago

tadhg-moore commented 4 years ago

image

When accessing data from netCDF files for deep lakes (>50m), on Windows RAM appears to get overloaded despite the netCDF files being relatively small (~40-80MB) and it causes my computer to freeze. This is not urgent but will require further investigation if this is a Windows issue, RStudio issue, R issue or maybe just a Tadhg issue.

tadhg-moore commented 4 years ago

If this is a depth issue i.e. outputting every 0.5m, then a potential solution could be providing a csv file with the list of depths to output the data at to prevent really high resolution to depths >50m

tadhg-moore commented 4 years ago

Upon further investigation, it is NOT a depth related issue but it is the number of depths in the netCDF file rather than the actual depth. For example, I ran this on a lake which had observations every mm so even though the lake was only 9.3m deep, there were 9287 unique depths in the netCDF file. So load_var, worked fine but we were passing many NAs to ggplot which was causing the RAM to explode. Will add a fix to my devel branch