search

Weiming-Hu / AnalogsEnsemble

The C++ and R packages for parallel ensemble forecasts using Analog Ensemble
https://weiming-hu.github.io/AnalogsEnsemble/
MIT License
18 stars 5 forks source link

Insufficient memory when dealing when large objects #7

Closed Weiming-Hu closed 5 years ago

Weiming-Hu commented 5 years ago

When dealing with large objects in R, the memory is exhausted.

> AnEn <- generateAnalogs(config)
Convert R objects to C++ objects ...
Computing standard deviation ... 
Computing mapping from forecast [Time, FLT] to observation [Time]  ... 
Computing search space extension ... 
Computing search windows for FLT ... 
Computing similarity matrices ... 
Error in .generateAnalogs(configuration$test_forecasts, dim(configuration$test_forecasts),  : 
  std::bad_alloc
Weiming-Hu commented 5 years ago

In commit 81dff38eb8d2ca6c5d88710bebf00ece930d51e1, I added a check for memory allocation success. However, with the current implementation to store all the similarity values, this is going to be a memory-intensive task.

Weiming-Hu commented 5 years ago

The memory usage is a linear relationship with the number of test days, search days. And it also depends on whether search stations are used and how many search stations are kept. The program will yield errors when memory is insufficient. The ticket is closed due to the user's perspective of this problem.