logpai / logparser

A machine learning toolkit for log parsing [ICSE'19, DSN'16]
Other
1.6k stars 551 forks source link

Spell optimization in python using multiprocessing #91

Closed sandyflute closed 1 year ago

sandyflute commented 1 year ago

Thanks for sharing this repo.

Has anyone tired optimizing the parsers ? Spell/drain ?

It is taking lot of time to create templates for large logs.

donvini94 commented 1 year ago

Seconding this, just adapted spell to my workflow and compared to my ibm/drain3 derived drain implementation it is insanely slow. Drain parsing: 5 minutes, Spell parsing 2h.

Not only is the spell implementation not optimized for multi-processors, there are also many expensive and inefficient python function calls that add up.

The whole LCS implementation is questionable and I wonder how they did their experiments for their tools and benchmarks paper.