magneticstain / Inquisition

An advanced and versatile open-source network anomaly detection platform
MIT License
8 stars 4 forks source link

Convert Destiny.getUniqueLogDataFields() To Static Function #68

Closed magneticstain closed 6 years ago

magneticstain commented 6 years ago

This is a Codacy recommendation and will make the function more clear to devs.

magneticstain commented 6 years ago

Will need to refactor function itself, references to it, and the unit tests for it.

magneticstain commented 6 years ago

No exceptions during test runs and unit tests are passing. Should be good now.

2017-11-25 12:13:27,452 [ DEBUG ] [ lib.destiny.Sage ] network threat engine is sleeping for [ 15 ] seconds before restarting routines
2017-11-25 12:13:42,465 [ INFO ] [ lib.destiny.Sage ] fetching intel, baseline, and log data (in that order) for threat detection model
2017-11-25 12:13:42,465 [ DEBUG ] [ lib.destiny.Sage ] fetching training intel data
2017-11-25 12:13:42,465 [ DEBUG ] [ lib.destiny.Sage ] fetching logs from log db
2017-11-25 12:13:42,465 [ DEBUG ] [ lib.destiny.Sage ] decoding all log data from log db
2017-11-25 12:13:44,067 [ DEBUG ] [ lib.destiny.Sage ] fetched and decoded [ 100 ] log records from db
2017-11-25 12:13:44,067 [ DEBUG ] [ lib.destiny.Sage ] fetching training baseline data
2017-11-25 12:13:44,067 [ DEBUG ] [ lib.destiny.Sage ] fetching logs from log db
2017-11-25 12:13:44,067 [ DEBUG ] [ lib.destiny.Sage ] decoding all log data from log db
2017-11-25 12:13:56,348 [ DEBUG ] [ lib.destiny.Sage ] fetched and decoded [ 71542 ] log records from db
2017-11-25 12:13:56,445 [ DEBUG ] [ lib.destiny.Sage ] fetching testing (raw log) data
2017-11-25 12:13:56,446 [ DEBUG ] [ lib.destiny.Sage ] fetching logs from log db
2017-11-25 12:13:56,446 [ DEBUG ] [ lib.destiny.Sage ] decoding all log data from log db
2017-11-25 12:14:03,971 [ DEBUG ] [ lib.destiny.Sage ] fetched and decoded [ 41270 ] log records from db
2017-11-25 12:14:04,011 [ DEBUG ] [ lib.destiny.Sage ] initializing intel data
2017-11-25 12:14:04,024 [ DEBUG ] [ lib.destiny.Sage ] calculating unique field list for training and testing data initialization
2017-11-25 12:14:04,216 [ DEBUG ] [ lib.destiny.Sage ] initializing log data matrix
2017-11-25 12:14:04,730 [ INFO ] [ lib.destiny.Sage ] vectorizing log data
2017-11-25 12:14:06,997 [ INFO ] [ lib.destiny.Sage ] training threat detection model
2017-11-25 12:14:08,252 [ INFO ] [ lib.destiny.Sage ] training complete; starting network threat analysis against current log data
2017-11-25 12:14:08,252 [ DEBUG ] [ lib.destiny.Sage ] initializing log data
2017-11-25 12:14:08,252 [ DEBUG ] [ lib.destiny.Sage ] initializing log data matrix
2017-11-25 12:14:08,632 [ INFO ] [ lib.destiny.Sage ] vectorizing log data
2017-11-25 12:14:09,521 [ INFO ] [ lib.destiny.Sage ] making predictions for testing data
2017-11-25 12:14:10,126 [ INFO ] [ lib.destiny.Sage ] threat detection results :: { [0 0 0 ..., 0 0 0] }
2017-11-25 12:14:10,126 [ DEBUG ] [ lib.destiny.Sage ] processing threat detection results
2017-11-25 12:14:10,128 [ DEBUG ] [ lib.destiny.Sage ] network threat engine is sleeping for [ 15 ] seconds before restarting routines
josh@jimmy1 ~/Code/Inquisition $ python3 -m unittest discover -s build/tests/ && python3 -m unittest discover -s build/test_TESTS/
...............................................Exception ignored in: <bound method Pygtail.__del__ of <pygtail.core.Pygtail object at 0x7fa365367ac8>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/pygtail/core.py", line 91, in __del__
    if self._filehandle():
  File "/usr/local/lib/python3.5/dist-packages/pygtail/core.py", line 173, in _filehandle
    self._fh = open(filename, "r", 1)
PermissionError: [Errno 13] Permission denied: '/var/log/inaccessible_test_log'
.Exception ignored in: <bound method Pygtail.__del__ of <pygtail.core.Pygtail object at 0x7fa3630dca20>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/pygtail/core.py", line 91, in __del__
    if self._filehandle():
  File "/usr/local/lib/python3.5/dist-packages/pygtail/core.py", line 173, in _filehandle
    self._fh = open(filename, "r", 1)
FileNotFoundError: [Errno 2] No such file or directory: '/var/log/non_existant_file'
....................
----------------------------------------------------------------------
Ran 68 tests in 36.582s

OK

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
josh@jimmy1 ~/Code/Inquisition $