search

Dunttus / AI-Project

Anomaly detection with Tensorflow.
GNU General Public License v3.0
7 stars 1 forks source link

Web access logs dataset #2

Open kosmylo opened 2 years ago

kosmylo commented 2 years ago

It is not clear to me where should I find the dataset to train the model. Should I have a website running with nginx or apache2? Is it possible to test with a public dataset?

Dunttus commented 2 years ago

Sorry that I haven't answered earlier, this old projekt that is not being maintained anymore. You need to train model with your own servers normal logs with no abnormal data for it to detect new anomalies in logs. Running train.py will only work with one log format that machine learning model was made with, more info in install page /AI-Project/install/README.md

Sure you can train a model with public dataset but it will detect everything as an anomaly since public dataset will most likely be very different from your own server logs.

kosmylo commented 2 years ago

Thanks for the information. As a follow-up question is how you test your model to find accuracy? What did you use as anomalies? What is the difference between TRAINING_DATA and MONITORED_LOG?