search

xrmx / bootchart

merge of bootchart-collector and pybootchartgui
GNU General Public License v2.0
233 stars 87 forks source link

_parse_proc_ps_log rewrite with iterator #60

Closed abone28 closed 9 years ago

abone28 commented 9 years ago

On my Ubuntu host with 2GiB of RAM rendering 100 seconds of bootchart data uses about 500-600 MiB of memory. This is too large for me.

I rewrite most memory hungry function _parse_proc_ps_log with iterators. New code uses only 100-150 MiB of memory. Hope you find this patch useful too.

xrmx commented 9 years ago

Thanks for your patch, will review it later today.

xrmx commented 9 years ago

On my ~850Kbs tgz we save 14% ram but we are ~22% slower (measured with /usr/bin/time) . Can you share your tarball to double check please?

abone28 commented 9 years ago

Sorry. My fault. I should not touch _parse_timed_block.

My host uses ubuntu version of bootchart which collect data via /proc interface. Here is my bootchart tarball. https://drive.google.com/file/d/0Bzi3TyA9Au_uaVF2N1IzUzFoVGM/view

I was not able to check patch influence on data collected via tasklog. I will rewrite the patch shortly and update pull request. Could you share your tarball so I can test it before commit.

abone28 commented 9 years ago

Reverted _parse_timed_blocks to old in-memory parsing. This should limit iterdecode performance penalty.

Parsing taskstat.log shold be as fast as in version 0.14.7.

Parsing proc_ps.log should be faster than previous commit, but still a bit slower than version 0.14.7. Memory footprint should be significantly lower than in version 0.14.7.

In the next plot (version 0.14.7 - black line, previous commit - blue line, current commit - red line)

mprof

xrmx commented 9 years ago

Thanks a lot Andrey! I've squashed in one commit and cherry-picked to master in 3ab81137cafe25c2ca4bc3a5f322a63646f9ce8d. While at it fixed it with python3 and added some numbers to the commit message.