search

nawendt / gribr

GRIB interface for R using ECMWF ecCodes
BSD 3-Clause "New" or "Revised" License
24 stars 1 forks source link

Memory management for batch processing a large number of grib files #10

Closed thanosk closed 6 years ago

thanosk commented 6 years ago

@nawendt Thank you for this project.

I would like your opinion when working with a large number of grib files (1000 files or more). When using a code like this:

myfiles <- list.files(mypath)

for (myfile in myfiles) {
  g <- grib_open(myfile)
  ...
  operations that utilize grib_list(), grib_get_message(), etc ...
  ...
  grib_close(g)
}

memory management in R becomes a problem. Although grib_close() is used, each iteration reserves a significant amount of memory which never gets freed leading to total memory exhaustion (24GB of RAM are not enough for processing ~1000 grib files, each one about 8kb). I tried using gc() and rm() for specific temporary variables after each for-loop iteration, but nothing changes. The only way to really free reserved memory is to end the R session.

Have you ever faced such problems and what would you propose as a workaround ? Thank you in advance.

nawendt commented 6 years ago

@thanosk No, I have not experienced this issue nor have I heard from anyone else who has this issue either. It's not a workflow that I did any testing with during any development. What you have tried already would be the first things I would have done myself. I will look into it and see if I can make sense of it.

nawendt commented 6 years ago

@thanosk I believe that I have found the issue. It appears I had not been as careful cleaning up some of the ecCodes objects (handles, iterators, etc.) and this was leading to the memory leaks that you were experiencing. The scale at which you are using the package is likely the reason why things became so noticeable and why I missed it with much smaller test cases. My testing with valgrind shows my fixes are working correctly, but I will do a few more tests to be sure. If all goes well, I should have a fixed version out fairly soon.

thanosk commented 6 years ago

Thank you so much for your immediate response and resolving. It behaves like a charm now! ;-)