search

logpai / Drain3

A robust streaming log template miner based on the Drain algorithm
Other
458 stars 131 forks source link

TensorFlow variant #39

Open sdwalker62 opened 3 years ago

sdwalker62 commented 3 years ago

Good afternoon!

Our ML team uses drain3 to transform system logs as part of a larger classification pipeline. In this pipeline, we use a pre-trained template miner to transform all of the batched logs being passed into the classifier for training. We are currently investigating how this could be done using tf.data.Dataset.map API to keep the pipeline efficient.

To this end, we were curious if any other drain3 users could benefit from a TensorFlow variant of the TemplateMiner. We have experience with TensorFlow and drain3 and would be willing to begin work on such a project.

davidohana commented 3 years ago

I think this might be a great contribution. Can you outline the benefits you see in a TensorFlow variant? Will it be able to run on GPU? Do you plan to design it as a standalone repo or as an addition to this repo? Will you re-use or reference the existing Drain3 code or rewrite it?

Superskyyy commented 1 year ago

We are implementing Drain3 on Ray and Flink and it works well. They should greatly enhance the performance of drain deployments. Although I'm not sure what a deep learning framework based Drain3 will bring to the users, since it will be hard to accelerate anything as there's no matrix or tensor operations required.