Anomaly Detection Engine for Linux Logs (ADE)

ADE can process a large numbers of logs from a large number of Linux systems to create a compact summary of those logs. The summary identifies and consolidates similar text strings into a single message example and assigns it a key (message id).  The summary determines if  the message id are being issued when expected, are being issued at the expected rate during a time slice, and how often during the day are the message or a similar message (same message id) issued.

You can use those results to examine

• A set of logs to find anomalies which may be helpful when attempting to find the root cause of a problem or incident
• The currently generated logs to find anomalies which may be helpful when attempting to find the cause of an on-going problem or incident

Please see http://openmainframeproject.github.io/ade/ for documentation on ADE.

Releases

Saw Kill 1.0.4

• Fix problems with timezone in Junit test
• Fix problems with year of test data in function test

Fall Kill 1.0.3

• Support for Core Infrastructure Initiative
  • add travis-ci build for every pull request
  • analyze every pull request with Sonarqube
  • store results of Sonarqube analysis at Sonarqube.com
• Add sample to mask sensitive data within Linux logs to allow sharing of logs
• Fix problem with train_test.sh
• Fix additional problems identified by Sonarqube

Poesten Kill 1.0.2

• Support for changing analytics
  • command to check syntax of model (flowlayout.xml file)
  • command to print out statistical information contained within model file (.bin file) to text file
  • command to print out version of code and data base
• Multiple SonarQube(TM) issues fixed
• Fix to problem with regression test
• Wiki article "Example of reading ADE data into R objects"

Esopus Creek 1.0.1

• Support for MariaDB(TM)
• Verify script - determine if sufficient messages are available to create a valid model
• Multiple SonarQube(TM) issues fixed
• Wiki article "Hints on how to update XSLT - tailor the output shown in a browser to problem"

Initial release 1.0.0

• Parsing of Linux Logs in RFC5424 and RFC3164 format
• Splitting logs into time slices
• Handling wrapper messages
• Statistical analysis of logs
• Creates output