mongodb-labs / edda

A log visualizer for MongoDB - This Repository is NOT a supported MongoDB product
232 stars 28 forks source link

Improve speed on long logs #75

Closed kchodorow closed 12 years ago

kchodorow commented 12 years ago

On the file Kaushal has.

samantharitter commented 12 years ago

-one optimization is to order the filters by how commonly they find matches, because lines are only parsed until the first time they match a filter.

kaushal commented 12 years ago

Based on the logs that I had to test with, I found that the following order of occurrence is most common when it comes to messages: rs_status fsync_lock rs_sync init_and_listen stale_secondary rs_exit rs_reconfig

Perhaps I need to put stale secondary later, but that shouldn't be too much of an issue. I changed the order in which the files are parsed, and while I can't see a noticable difference in the log files I have at my disposal, the program is actually doing fewer operations.

kchodorow commented 12 years ago

This needs more analysis, the current speed is not good enough. Try running a code profiler on edda and figure out where it's spending most of its time.

kaushal commented 12 years ago

On the 780 MB log file you gave us our time improved from about 3:40 to about 0:40, which makes close to a 500% increase in efficiency, 500% decrease in in-efficiency? Either way, I am calling this issue closed.

kchodorow commented 12 years ago

Nice job.