search

ubarsc / pylidar

A set of Python modules which makes it easy to write lidar processing code in Python
http://www.pylidar.org
GNU General Public License v3.0
15 stars 11 forks source link

Possible memory leak in interpolation #4

Closed gillins closed 7 years ago

gillins commented 7 years ago

Original report by Neil Flood (Bitbucket: neilflood, GitHub: neilflood).

This probably should be an issue under pynninterp, but here will do for now.

Nick has some evidence to suggest a memory leak when running interpolations. He was running something Jaz set up to do interpolations across a large number of tiles (with a suitably small window size), and it grew and grew, to many Gb. This suggests that there is a memory leak, with something not being freed up between windows.

I should try to set up a suitable test loop over the guts of pynninterp, but wanted to log this here, so others might be able to look into it, too.

gillins commented 7 years ago

Original comment by Sam Gillingham (Bitbucket: gillins, GitHub: gillins).

I've just pushed up a test directory and main program to the pynninterp. Seems ok when run through valgrind. Was he running linear or natural neighbour?

We might need to grab an example of the points he was using - is possible the leak only happens in certain situations.

gillins commented 7 years ago

Original comment by Neil Flood (Bitbucket: neilflood, GitHub: neilflood).

Further checking with Nick now suggests that the problem is not a memory leak, but due to the combination of a large number of input files with high variability of point density across the whole region, so that for certain windows, the required memory can still be surprisingly large, compared to that required for other parts of the same job.