Perform a comprehensive literature study of network anomaly detection models in large scale enviornments

[sameeravithana]

Tasks DoD:

• [ ] Literature study on network anomaly detection models that were open-source and deployed in large scale
• [ ] Introduce new direction of machine learning models which OpenADS can co-relate with
• [ ] Document at project wiki

[xiaoleihuang]

I read some papers. Most common methods deploy clustering based methods. The statistical learning might not be useful sometime though.

Here is a paper that introduces some anomaly detection open source softwares. Surveys.pdf

I will add more comments if possible. But currently I am overwhelmed by my master thesis's deadline this month.

[xiaoleihuang]

I found some open source resources might be useful. Here is a list of machine learning open sources overall: http://jmlr.org/mloss/; I search some open sources that support large scale machine learning methods:

1. Mahout http://mahout.apache.org/. Yes, mahout is known for recommender system, it is also known for optimizing math computation like linear algebra. It is powerful tool for its built-in ML packages.
2. Spark Mlib: http://mlbase.org/ This contains a collections of ML algorithms and models that can be deployed on Spark.
3. Others such as H2O https://github.com/h2oai/h2o-2

For Anomaly Detection, Twitter owns one: https://github.com/twitter/AnomalyDetection For streaming anomaly detection evaluation, this link will be helpful: https://github.com/numenta/nab

[xiaoleihuang]

New direction might be Deep Learning, such as Recurrent or Convolutional Neural Networks, LSTM, etc. But those are computationally expensive. Open Sources: Theano, TensorFlow, Deeplearning4j(java).

[Ammoniya]

archive