Inspect App Memory Usage

[Kovak]

We need to inspect the memory usage to determine if and where optimizations and clean up need to be done

[sbrother]

Done and on branch 25-memory-profiler-a. Copying relevant email below:

I've found a really nice, fast and easy to use python memory profiler and added it to the branch 25-memory-profiler-a. You'll have to install it on each computer you use, so I provided a script (install_memory_profiler.sh) that will install the required modules. It requires sudo permissions.

Once it's installed, you need to tell it what function(s) you want to track the memory usage of. This is super easy: just go into main.py and add the decorator @profile in front of any def. This will produce a real-time line-by-line memory report of the function when you run main.py. I've added this to the PlayerCharacter _advance_time function just as an example; there are no memory leaks there (yay!) but we should probably inspect most of the code. You can inspect as many at a time as you want, but the reports are pretty detailed and this could get unwieldy pretty fast. If it's producing too much text, try piping the output to a file with python main.py > log.txt or something like that.

Hope this helps, looking forward to testing some hypotheses with hard data!

[sbrother]

I am extending this a bit prior to our meeting to make a "smarter" tool that can actually zero in on where memory is being leaked without requiring any manual work. Documenting that process here (i'll put it all into a script when I'm done)

1. add @profile at the right tablevel before every method: run this regex through sed: s|^(.*)(def [a-zA-Z].+)$|\1@profile\n\1\2|g

[sbrother]

OK, I've finished an automatic tool to find memory leaks. To run it run the shell script run_memory_profiler.sh. This will generate a huge fucking logfile named memlog.txt. You can then parse this log file by running _python parselogs.py memlog.txt

[Kovak]

Hey Sam,

Can we also make the automatic tool report the total time it ran as well as the total memory usage of the application.

This would help us distinguish which branches were bigger memory hogs than other branches.

Also we should probably look into getting KB instead of MB as our base data unit

[Kovak]

also could we maybe print the function or class name next to the line number, that'd be useful for making the tool a little bit more human readable so for instance I could ignore: that's just animation controller loading up assets and so on

[Kovak]

Memory usage is calculated into MB in this file: https://github.com/fabianp/memory_profiler/blob/master/memory_profiler.py

using 1 of 2 _get_memory methods found between lines 23 and 53.

It seems like we could remove the / (1024 \ 2) or alter it in order to change units in the line

26: mem = float(process.get_memory_info()[0]) / (1024 \ 2)

and

48: return float(out[1].split()[vsz_index]) / 1024

[sbrother]

You are the man. I will implement this when I get done with work today. Great find.

On Mon, Jan 14, 2013 at 3:18 PM, Kovak notifications@github.com wrote:

Memory usage is calculated into MB in this file: https://github.com/fabianp/memory_profiler/blob/master/memory_profiler.py

using 1 of 2 _get_memory methods found between lines 23 and 53.

It seems like we could remove the / (1024 \ 2) or later it in order to change units in the line

26: mem = float(process.get_memory_info()[0]) / (1024 \ 2)

and

48: return float(out[1].split()[vsz_index]) / 1024

— Reply to this email directly or view it on GitHubhttps://github.com/Kovak/CBL_SECRET_PROTOTYPE_RUNNING_GAME/issues/25#issuecomment-12245524.

[Kovak]

Thanks Sam

Ooo: consider adding the starting total memory usage and then the final total memory usage to the parser output as well as the total time running. Our big goal is to essentially have very little to no growth in memory usage over lifetime of the app independent of time. Or at least a known lower and upper bound for memory usage. We just don't want constant growth over time