search

jdkloe / pybufr-ecmwf

a python module that allows reading and writing BUFR formatted files, where BUFR stands for Binary Universal Form for the Representation of meteorological data.
Other
29 stars 12 forks source link

Big count of used memory #19

Closed DivinytyToffee closed 5 years ago

DivinytyToffee commented 5 years ago

Library use many RAM. Sometimes server crashed, when library use more 6 Gb, it's server ram limit. Is it normal for a library to consume so many resources or can it be somehow fix?

backend_1          | 2019-04-16T10:23:13.113753044Z     from pybufr_ecmwf.bufr import BUFRReader
backend_1          | 2019-04-16T10:23:13.113760127Z   File "/usr/local/lib/python3.7/site-packages/pybufr_ecmwf/bufr.py", line 40, in <module>
backend_1          | 2019-04-16T10:23:13.113765844Z     from .bufr_interface_ecmwf import BUFRInterfaceECMWF
backend_1          | 2019-04-16T10:23:13.113778707Z   File "/usr/local/lib/python3.7/site-packages/pybufr_ecmwf/bufr_interface_ecmwf.py", line 52, in <module>
backend_1          | 2019-04-16T10:23:13.113787664Z     raise e
backend_1          | 2019-04-16T10:23:13.113795684Z   File "/usr/local/lib/python3.7/site-packages/pybufr_ecmwf/bufr_interface_ecmwf.py", line 48, in <module>
backend_1          | 2019-04-16T10:23:13.113804747Z     from . import ecmwfbufr
backend_1          | 2019-04-16T10:23:13.113813227Z ImportError: Error loading shared library /usr/local/lib/python3.7/site-packages/pybufr_ecmwf/ecmwfbufr.cpython-37m-x86_64-linux-gnu.so: Out of memory
jdkloe commented 5 years ago

The library itself is only just above 1 MB after compilation on my system so this cannot be an explanation I think. Can you check the size of the file pybufr_ecmwf/ecmwfbufr.so on your machine?

Some other problem maybe present which prevents python to load the module. Can you check the memory usage (with a tool like top or similar) and then launch the decoding? What do you see? Other memory leaks may cause the same effect but actually be present in another script or program.

DivinytyToffee commented 5 years ago

Sorry for trouble you. We find leaks, that be in influxdb.

jdkloe commented 5 years ago

thanks for reporting back. I will close this issue now.