matsim-vsp / matsim-python-tools

Tools for working with matsim in python
58 stars 20 forks source link

Fix memory leak in ElementTree.iterparse #15

Closed ksauzz closed 2 years ago

ksauzz commented 2 years ago

Hi, I found a memory leak issue, and this PR fixes it.

Background

I tried to read a huge output_event.xml.gz over 3GB using event reader, then OOM killer killed the process on my machine which has 32GB RAM. After the investigation I found memory leak of the process using memory profiler.

Memory usage

Before

mprof-matsim

After the fix

mprof-fixed-matsim

the test code (memleak.py)

#!/usr/bin/env python                                                                                                                                                                                                                                                                       
import matsim                                                                                                                                                                                                                                                                               
from memory_profiler import profile                                                                                                                                                                                                                                                         
import xml.etree.ElementTree as ET

event_file = 'tmp/simulation/output_events.xml.gz'                                                                                                                                                                                                                                          
cnt = 1000000                                                                                                                                                                                                                                                                               

@profile                                                                                                                                                                                                                                                                                    
def run_matsim():                                                                                                                                                                                                                                                                           
    i = 0                                                                                                                                                                                                                                                                                   
    for event in matsim.event_reader(event_file):                                                                                                                                                                                                                                           
        i += 1                                                                                                                                                                                                                                                                              
        if i > cnt:                                                                                                                                                                                                                                                                         
            break                                                                                                                                                                                                                                                                           

show a memory usage graph

mprof run ./memleak.py        # run w/ profile
mprof plot                 # make a graph
ksauzz commented 2 years ago

Thank you!