recog_standardize: handle missing files, detect removed identifiers, unify device type tracking

TomSellers commented 2 years ago

Description

This PR fixes #384 and makes a few other changes.

Changes

Adds error handling so that if a particular identifiers file is missing the script will return an empty string so that the contents can be populated from the existing fingerprint database corpus.
Adds logic to detect and notify the user when values have been removed from an indicators list. This is useful when performing cleanups or widespread changes of values. No flag is required to enable this behavior, it works just like the previous detection of new values.
Unifies *.device identifiers in indicators/device.txt and removes the existing os.device and hw.device files. This has already helped identify a few issues which I have addressed.

Example output after deleting 3 valid entries from the device.txt and adding one bonus entry of Sample Device Type Thingy.

$ ruby bin/recog_standardize xml/*.xml -w
DEVICE REMOVED VALUE: Sample Device Type Thingy
DEVICE NEW VALUE: DSL Modem
DEVICE NEW VALUE: DSLAM
DEVICE NEW VALUE: DSU/CSU
$ echo $?                                
1

$ ruby bin/recog_standardize xml/*.xml   
$ echo $?                             
0

How Has This Been Tested?

ruby bin/recog_standardize xml/*.xml and reviewing the changes

Types of changes

Bug fix (non-breaking change which fixes an issue)

Checklist:

[x] I have updated the documentation accordingly (or changes are not required).
[x] I have added tests to cover my changes (or new tests are not required).
[x] All new and existing tests passed.

mkienow-r7 commented 2 years ago

What do you think about changes that would be compatible with the recog_standardize use in tools/dev/hooks/pre-commit? The new reset flag won't provide a focused alert for the user.

TomSellers commented 2 years ago

@mkienow-r7 - I've reworked this PR so that it should work just fine with the commit hook. Now detection of values that need to be removed is handled on every run and a console line which includes the text REMOVED VALUE is emitted. I've reworked this description.

TomSellers commented 2 years ago

@mkienow-r7 I went ahead and merged upstream/main into this PR to ensure that there were no issues. Glad to see master -> main, thanks for doing that.

rapid7 / recog