search

ait-aecid / logdata-anomaly-miner

This tool parses log data and allows to define analysis pipelines for anomaly detection. It was designed to run the analysis with limited resources and lowest possible permissions to make it suitable for production server use.
GNU General Public License v3.0
67 stars 23 forks source link

Parser count report all parsed lines #588

Closed landauermax closed 3 years ago

landauermax commented 3 years ago

Currently it is necessary to set the paths of the parser count. In many cases it is sufficient to just report the number of processed lines in total, not by specific paths. Especially, when someone just wants to check if the aminer works and there are logs in the file, but does not want to see the parser to find an usable path. I would propose that setting no paths (i.e., leaving out this parameter in the config) will result that the total number of processed lines is reported.

ernstleierzopf commented 3 years ago

It is not necessary to set the paths of parser count. Please see demo/aminer/demo-config.yml. There no path is configured for the ParserCount. An example output looks like this (from the demo):

Count report (1 lines)
  Parsed paths in the last 10 seconds:
    /model/ECD/a: {'CurrentProcessedLines': 1667, 'TotalProcessedLines': 1667}
    /model/ECD/b: {'CurrentProcessedLines': 1667, 'TotalProcessedLines': 1667}
    /model/ECD/c: {'CurrentProcessedLines': 1667, 'TotalProcessedLines': 1667}
    /model/ECD/d: {'CurrentProcessedLines': 1667, 'TotalProcessedLines': 1667}
    /model/ECD/e: {'CurrentProcessedLines': 1666, 'TotalProcessedLines': 1666}
    /model/ECD/f: {'CurrentProcessedLines': 1666, 'TotalProcessedLines': 1666}
    /model/DailyCron: {'CurrentProcessedLines': 66, 'TotalProcessedLines': 66}
    /model/Random: {'CurrentProcessedLines': 3165, 'TotalProcessedLines': 3165}
    /model/RandomTime: {'CurrentProcessedLines': 2216, 'TotalProcessedLines': 2216}
    /model/DiskReport: {'CurrentProcessedLines': 4, 'TotalProcessedLines': 4}
    /model/IPAddresses: {'CurrentProcessedLines': 6, 'TotalProcessedLines': 6}
    /model/type/syscall: {'CurrentProcessedLines': 11, 'TotalProcessedLines': 11}
    /model/type/path: {'CurrentProcessedLines': 10, 'TotalProcessedLines': 10}
    /model/CronAnnouncement: {'CurrentProcessedLines': 1, 'TotalProcessedLines': 1}

I see no problem with this output as it shows the total log lines for every path (the user should be able to calculate the total lines for all paths.) Please reopen if something in the ParserCount should be changed.

ernstleierzopf commented 3 years ago

i noticed that the CurrentProcessedLines are not reset to 0. This needs to be fixed.