search

magneticstain / Inquisition

An advanced and versatile open-source network anomaly detection platform
MIT License
8 stars 4 forks source link

Log Matches Are Redacted Regardless of Config Value #135

Closed magneticstain closed 5 years ago

magneticstain commented 5 years ago

There is a config value - printMatchValues - that allows an admin to specify whether matched values (and the source logs themselves) should be written to the app log. This setting does not appear to be working; inquisition.py always prints the matches, and never prints the logs, regardless of the config val.

The line appears to have been first added w/ commit https://github.com/magneticstain/Inquisition/commit/136327e84bf1821d516d0cbfd3afedbc3c681f1a and then updated to its current version w/ commit https://github.com/magneticstain/Inquisition/commit/869da63edf08003507e6ed26357dc2eba0a58ef1 .

magneticstain commented 5 years ago

Fixed:

sysadmin@lhr1inquisition01:~$ tail -F /var/log/inquisition/app.log | egrep 'processing log|POST-PROCESSED LOG|MATCHED|\]\]\]'
2018-11-18 03:43:37,240 [ DEBUG ] [ lib.anatomize.Parser ] processing log [[[ Nov 18 03:43:35 lhr1inquisition01 ntpd[724]: Soliciting pool server 85.199.214.222
 ]]] using [ PARSER ID: 1 // NAME: parser_system_logs // READING FROM LOG FILE: /var/log/syslog // OFFSET FILE: /opt/inquisition/tmp/1_parser_system_logs.offset // TOTAL LOGS PROCESSED: { 12 } ]
2018-11-18 03:43:37,240 [ DEBUG ] [ lib.anatomize.Parser ] POST-PROCESSED LOG [[[ Nov 18 03:43:35 lhr1inquisition01 ntpd[724]: Soliciting pool server 85.199.214.222
 ]]] using [ PARSER ID: 1 // NAME: parser_system_logs // READING FROM LOG FILE: /var/log/syslog // OFFSET FILE: /opt/inquisition/tmp/1_parser_system_logs.offset // TOTAL LOGS PROCESSED: { 12 } ]
2018-11-18 03:43:37,241 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 1 // NAME: template_timestamp // FIELD: timestamp // REGEX: {{ ^[A-Za-z]{3} [\d ]{2} [\d:]{8} }} // GRP: { 0 } ||  MATCH_IDX: { 0 } ] :: [ VALUE: Nov 18 03:43:35 ]
2018-11-18 03:43:37,242 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 2 // NAME: template_host_device // FIELD: host // REGEX: {{ [A-Za-z0-9]+ }} // GRP: { 0 } ||  MATCH_IDX: { 5 } ] :: [ VALUE: lhr1inquisition01 ]
2018-11-18 03:43:37,243 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 3 // NAME: template_app_name // FIELD: application // REGEX: {{ ([A-Za-z0-9]+)(\[[0-9]{1,6}\]:) }} // GRP: { 0 } ||  MATCH_IDX: { 0 } ] :: [ VALUE: ntpd ]
2018-11-18 03:43:37,244 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 4 // NAME: template_linux_log_msg // FIELD: message // REGEX: {{ (\[[0-9]{1,6}\]: )([\S\t ]+) }} // GRP: { 1 } ||  MATCH_IDX: { 0 } ] :: [ VALUE: Soliciting pool server 85.199.214.222 ]
2018-11-18 03:43:39,351 [ DEBUG ] [ lib.anatomize.Parser ] processing log [[[ Nov 18 03:43:38 lhr1inquisition01 sshd[29877]: Connection closed by 67.175.207.185 port 40858 [preauth]
 ]]] using [ PARSER ID: 2 // NAME: parser_auth_log // READING FROM LOG FILE: /var/log/auth.log // OFFSET FILE: /opt/inquisition/tmp/2_parser_auth_log.offset // TOTAL LOGS PROCESSED: { 18 } ]
2018-11-18 03:43:39,352 [ DEBUG ] [ lib.anatomize.Parser ] POST-PROCESSED LOG [[[ Nov 18 03:43:38 lhr1inquisition01 sshd[29877]: Connection closed by 67.175.207.185 port 40858 [preauth]
 ]]] using [ PARSER ID: 2 // NAME: parser_auth_log // READING FROM LOG FILE: /var/log/auth.log // OFFSET FILE: /opt/inquisition/tmp/2_parser_auth_log.offset // TOTAL LOGS PROCESSED: { 18 } ]
2018-11-18 03:43:39,353 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 1 // NAME: template_timestamp // FIELD: timestamp // REGEX: {{ ^[A-Za-z]{3} [\d ]{2} [\d:]{8} }} // GRP: { 0 } ||  MATCH_IDX: { 0 } ] :: [ VALUE: Nov 18 03:43:38 ]
2018-11-18 03:43:39,354 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 2 // NAME: template_host_device // FIELD: host // REGEX: {{ [A-Za-z0-9]+ }} // GRP: { 0 } ||  MATCH_IDX: { 5 } ] :: [ VALUE: lhr1inquisition01 ]
2018-11-18 03:43:39,355 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 3 // NAME: template_app_name // FIELD: application // REGEX: {{ ([A-Za-z0-9]+)(\[[0-9]{1,6}\]:) }} // GRP: { 0 } ||  MATCH_IDX: { 0 } ] :: [ VALUE: sshd ]
2018-11-18 03:43:39,356 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 4 // NAME: template_linux_log_msg // FIELD: message // REGEX: {{ (\[[0-9]{1,6}\]: )([\S\t ]+) }} // GRP: { 1 } ||  MATCH_IDX: { 0 } ] :: [ VALUE: Connection closed by 67.175.207.185 port 40858 [preauth] ]
^C
sysadmin@lhr1inquisition01:~$ grep printMatchValues /opt/inquisition/conf/main.cfg 
printMatchValues = 1
sysadmin@lhr1inquisition01:~$ 
sysadmin@lhr1inquisition01:~$ tail -F /var/log/inquisition/app.log | egrep 'processing log|POST-PROCESSED LOG|MATCHED'
2018-11-18 03:38:11,573 [ DEBUG ] [ lib.anatomize.Parser ] processing log [[[ < REDACTED BY CONFIG > ]]] using [ PARSER ID: 1 // NAME: parser_system_logs // READING FROM LOG FILE: /var/log/syslog // OFFSET FILE: /opt/inquisition/tmp/1_parser_system_logs.offset // TOTAL LOGS PROCESSED: { 37 } ]
2018-11-18 03:38:11,573 [ DEBUG ] [ lib.anatomize.Parser ] POST-PROCESSED LOG [[[ < REDACTED BY CONFIG > ]]] using [ PARSER ID: 1 // NAME: parser_system_logs // READING FROM LOG FILE: /var/log/syslog // OFFSET FILE: /opt/inquisition/tmp/1_parser_system_logs.offset // TOTAL LOGS PROCESSED: { 37 } ]
2018-11-18 03:38:11,574 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 1 // NAME: template_timestamp // FIELD: timestamp // REGEX: {{ ^[A-Za-z]{3} [\d ]{2} [\d:]{8} }} // GRP: { 0 } ||  MATCH_IDX: { 0 } ]
2018-11-18 03:38:11,575 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 2 // NAME: template_host_device // FIELD: host // REGEX: {{ [A-Za-z0-9]+ }} // GRP: { 0 } ||  MATCH_IDX: { 5 } ]
2018-11-18 03:38:11,576 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 3 // NAME: template_app_name // FIELD: application // REGEX: {{ ([A-Za-z0-9]+)(\[[0-9]{1,6}\]:) }} // GRP: { 0 } ||  MATCH_IDX: { 0 } ]
2018-11-18 03:38:11,577 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 4 // NAME: template_linux_log_msg // FIELD: message // REGEX: {{ (\[[0-9]{1,6}\]: )([\S\t ]+) }} // GRP: { 1 } ||  MATCH_IDX: { 0 } ]
2018-11-18 03:38:15,583 [ DEBUG ] [ lib.anatomize.Parser ] processing log [[[ < REDACTED BY CONFIG > ]]] using [ PARSER ID: 1 // NAME: parser_system_logs // READING FROM LOG FILE: /var/log/syslog // OFFSET FILE: /opt/inquisition/tmp/1_parser_system_logs.offset // TOTAL LOGS PROCESSED: { 38 } ]
2018-11-18 03:38:15,584 [ DEBUG ] [ lib.anatomize.Parser ] POST-PROCESSED LOG [[[ < REDACTED BY CONFIG > ]]] using [ PARSER ID: 1 // NAME: parser_system_logs // READING FROM LOG FILE: /var/log/syslog // OFFSET FILE: /opt/inquisition/tmp/1_parser_system_logs.offset // TOTAL LOGS PROCESSED: { 38 } ]
2018-11-18 03:38:15,585 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 1 // NAME: template_timestamp // FIELD: timestamp // REGEX: {{ ^[A-Za-z]{3} [\d ]{2} [\d:]{8} }} // GRP: { 0 } ||  MATCH_IDX: { 0 } ]
2018-11-18 03:38:15,586 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 2 // NAME: template_host_device // FIELD: host // REGEX: {{ [A-Za-z0-9]+ }} // GRP: { 0 } ||  MATCH_IDX: { 5 } ]
2018-11-18 03:38:15,586 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 3 // NAME: template_app_name // FIELD: application // REGEX: {{ ([A-Za-z0-9]+)(\[[0-9]{1,6}\]:) }} // GRP: { 0 } ||  MATCH_IDX: { 0 } ]
2018-11-18 03:38:15,587 [ DEBUG ] [ lib.anatomize.Parser ] template MATCHED log :: [ TID: 4 // NAME: template_linux_log_msg // FIELD: message // REGEX: {{ (\[[0-9]{1,6}\]: )([\S\t ]+) }} // GRP: { 1 } ||  MATCH_IDX: { 0 } ]
^C
sysadmin@lhr1inquisition01:~$ grep printMatchValues /opt/inquisition/conf/main.cfg 
printMatchValues = 0
sysadmin@lhr1inquisition01:~$