search

Thijsvanede / DeepLog

PyTorch implementation of Deeplog: Anomaly detection and diagnosis from system logs through deep learning
MIT License
199 stars 48 forks source link

How to convert log events into numbers #8

Closed alishan2040 closed 2 years ago

alishan2040 commented 2 years ago

Hi, thanks for providing this implementation. I had found a number of deeplog implementations online but this one seems more suitable for my case. I've a log file which only contain names of the events and machine which produced. For example;

bot.init.started, bot1 bot.connection.established, bot1 bot.message.sent, bot1 ... I want to train deeplog on the normal execution of the bot while detect anomalies on the abnormal sequence of execution. As there are 9 different types of events produced by a single machine, I marked them from 0 to 8 in a single session. Is this approach going to work or have to extract templates? Is there a proper method which transforms these event ids into numbers for training deeplog. I'm not clear as most of the deeplog implementations are very specific to certain type of logs such as HDFS.

Thanks.

Thijsvanede commented 2 years ago

Hello,

I am glad you liked our implementation.

Manually transforming these logs will certainly work. We do provide a preprocessor that performs this process automatically: Preprocessor.csv. However, our preprocessor requires there to be timestamps within the csv file, so you would have to add them yourself. If they are already in order, I believe you can have simple incremental timestamps (e.g., 0, 1, 2, ...) or simply set all timestamps to 0.