Improve performance of cmd_print

gforcada / haproxy_log_analysis

HAProxy log analyzer

https://pypi.org/project/haproxy_log_analysis

GNU General Public License v3.0

89 stars 35 forks source link

Improve performance of cmd_print #19

Closed pts-kevinqiu closed 7 years ago

pts-kevinqiu commented 8 years ago

My log file has about 25k log entries after filtering. With the previous implementation, cmd_print hangs for a long time with CPU at 100%.

Using for loop and concate a string is expensive, since the previous data object is thrown away and may get GC'ed.

Switched to a list comprehension and this function finished in 0.01 second (on pypy, haven't tested it on cpython).

gforcada commented 8 years ago

@pts-kevinqiu Wow cool! Thanks for looking into this. The only performance related action I did on the code is creating the pickle files, I never checked the commands/filters performance.

Two things before I merge this pull request:

would you mind squashing together the last 3 commits? So that the pull request only has two commits
please add a change log entry on CHANGES.rst

Thanks!!

coveralls commented 7 years ago

Coverage remained the same at 100.0% when pulling 660004ee2c8b9db9e5220e1e75759d833b66e0d2 on pts-kevinqiu:fix-cmd-print-performance-issue into 2190a50bfc4529cab3ff62d8bd67b2f49e1eb6b6 on gforcada:master.

pts-kevinqiu commented 7 years ago

@gforcada Done :)

gforcada commented 7 years ago

@pts-kevinqiu and haproxy_log_analysis 2.0.2 is out! Thanks a lot!