search

NCAR / pynio

PyNIO is a multi-format data I/O package with a NetCDF-style interface
http://www.pyngl.ucar.edu/Nio.shtml
Apache License 2.0
112 stars 37 forks source link

GRIB2 memory leak #21

Closed wmay closed 5 years ago

wmay commented 6 years ago

There appears to be a memory leak when opening GRIB2 files.

Example

while True:
    ds = Nio.open_file('ruc2_130_20110624_0100_000.grb2')
    ds.close()

In this example the amount of memory used on my computer increases by about 9MB each iteration. (Download the example file here.)

I found a discussion on the NCL mailing list about a memory leak with GRIB2 files, and it was apparently fixed last year. If this is related to the PyNIO memory leak, maybe it can be fixed just by updating the NCL files?

bladwig1 commented 6 years ago

Those NCL changes haven't been integrated with PyNIO yet, mainly due to the large amount of changes to the HDF5 code, which likely requires a bit of work for the PyNIO wrappers. I suspect that is probably the cause as well. It will be updated eventually, but I don't have an ETA at this time. Thank you for the file!

If you feel like living dangerously, there's an 'ncl-update' branch that contains the new NCL updates, but it won't have the appropriate Python wrappers that support Python 3.x or the new HDF5 updates. You might be able to pull the GRIB stuff out of 'ncl-update' and put them in the 'develop' branch, rebuild, and it'll magically work. See build_py2.sh and build_py3.sh for some scripts to help with building (using a conda environment with the dependencies already installed). If you should venture down this dark and dangerous path and it fixes your problem, please let us know!

wmay commented 6 years ago

Well I can confirm that the issue is fixed in the ncl-update branch. It'll take me a little more time to try combining the branches.

rcabell commented 5 years ago

Merging the code from the two branches works great and lets me use Python 3 and GRIB2 without memory blowing up.