'TypeError' while comparing Detection and Visibility Coverage

[robotbond]

Command : python3 dettect.py v -ft ABC/techniques-administration-empty-data-source-admin-file-windows-linux-aws-azure-office-365(9).yaml -fd ABC/data-sources-diageo2.yaml -o Error: Traceback (most recent call last): File "dettect.py", line 309, in _menu(_init_menu()) File "dettect.py", line 233, in _menu generate_visibility_layer(file_tech, args.file_ds, True) File "/home/admin1/DeTTECT/technique_mapping.py", line 46, in generate_visibility_layer mapped_techniques_both = _map_and_colorize_techniques_for_overlaid(my_techniques, my_data_sources, platform) File "/home/admin1/DeTTECT/technique_mapping.py", line 303, in _map_and_colorize_techniques_for_overlaid tcnt = len([d for d in technique_data['detection'] if get_latest_score(d) >= 0]) File "/home/admin1/DeTTECT/technique_mapping.py", line 303, in tcnt = len([d for d in technique_data['detection'] if get_latest_score(d) >= 0]) TypeError: '>=' not supported between instances of 'NoneType' and 'int'

[rubinatorz]

hi Nitin,

That error raises because one of the score entries of the detection node in de techniques YAML file is not a valid number (between -1 and 5). For example:

- technique_id: T1173
  technique_name: Dynamic Data Exchange
  detection:
    applicable_to: [all]
    location:
    - 'My SIEM'
    comment: ''
    score_logbook:
    - date: 2020-05-11
      score: null
      comment: ''
...

This score of null will result in the error you got. Maybe there are more invalid values. So I would suggest to look for detection scores that are not a valid number between -1 and 5.

Regards, Ruben

[robotbond]

Hello Ruben,

I don't find the error as all detection score lies between -1 and 5. How to troubleshoot the issue ?

Regards, Nitin

Regards, Nitin

[rubinatorz]

Hi Nitin,

An easy way to find out in which technique the error resides, is by adding the following line to technique_mapping.py just above line 303:

print(technique_id)

Run your dettect command again. A list with technique ID's will be printed and the last technique ID is the one with the error in the score.

Please let us know if you found it, or share the YAML portion of the technique with the error (please remove any sensitive information).

Regards, Ruben

[robotbond]

Hi Ruben,

Please find the below output. FYI- I filled the technique file using DeTT&ct Editor so there is no way I could've given null or a wrong value.

Output T1001 T1002 T1003 T1004 T1005 Traceback (most recent call last): File "dettect.py", line 309, in _menu(_init_menu()) File "dettect.py", line 233, in _menu generate_visibility_layer(file_tech, args.file_ds, True) File "/home/admin1/DeTTECT/technique_mapping.py", line 46, in generate_visibility_layer mapped_techniques_both = _map_and_colorize_techniques_for_overlaid(my_techniques, my_data_sources, platform) File "/home/admin1/DeTTECT/technique_mapping.py", line 304, in _map_and_colorize_techniques_for_overlaid tcnt = len([d for d in technique_data['detection'] if get_latest_score(d) >= 0]) File "/home/admin1/DeTTECT/technique_mapping.py", line 304, in tcnt = len([d for d in technique_data['detection'] if get_latest_score(d) >= 0]) TypeError: '>=' not supported between instances of 'NoneType' and 'int'

Yaml portion of T1005 echnique_id: T1005 technique_name: Data from Local System detection:

• applicable_to:
  • all location:
  • '' comment: '' score_logbook: [] visibility:
• applicable_to:
  • all comment: score_logbook:
  • date: 2020-05-06T00:00:00.000Z score: 3 comment: '' auto_generated: true

Regards, Nitin

[rubinatorz]

Hi Nitin,

The problem is the empty score_logbook item under the detection node:

score_logbook: []

Try changing this to:

score_logbook:
    - date:
      score: -1
      comment: ''

And that should work. Or maybe more entries with empty score_logbook are there.

One other question: did you edit the YAML file in a text editor or did you use the DeTT&CT Editor?

Cheers, Ruben

[robotbond]

Hi Ruben,

I used the DeTT&ct editor and I left the detection score logbook as it is where detection is not there for the respective technique. Meaning the score_logbook set as -1 as default. There are many techniques for which I've not set the score logbook and left those as it is.

[rubinatorz]

Hi Nitin,

I will look into that, I guess the DeTT&CT Editor is not filling the score_logbook for detection and the CLI is not accepting this kind of YAML content. Give me some time to investigate it. I'll come back on this asap.

Regards, Ruben

[robotbond]

Sure Ruben, look forward for the fix. You guys are doing a fantastic job.

[rubinatorz]

Hi Nitin,

I investigated the issue and found the problem. The DeTT&CT Editor that was removing empty score_logbook entries (for sanity reasons), while it should keep 1 empty item. I fixed this in the master branch. If you are running the Editor locally, just do a git pull. I also deployed a new version of the online Editor at: https://rabobank-cdc.github.io/dettect-editor/. If it doesn't work directly, remove your browser cache because depending on your browser a lot of stuff is cached.

Regards, Ruben

[robotbond]

Thanks Ruben, it worked. Just one feedback, it would be great if we can have a range of Detection and Visibility scores in legends. For ex there is a legend for Detection and Visibility which is applied to a detection which has scored 2 out of 5. So if we can have range of Detection and Visibility scores in legends then it will be useful.

[marcusbakker]

Hi Nitin,

Could you explain this a bit further? The description per detection or visibility score is described on the Wiki, scoring table xlsx sheet and can be viewed in the Editor. Or is that not the kind of info you are looking for?

Regards, Marcus

[robotbond]

Hi Marcus, I referred to "Visibility and detection coverage" comparison where we have legends such as 'Visibility', 'Detection' and "Visibility and Detection". What I'm missing is to have range of score for "Visibility and Detection" so that we can identify the techniques for which we have low detection and visibility rate.

You will need to add two line items in the legends Detection score 4-5, Visibility score 4-5 | Detection and Visibility Detection score 1-3 Visibility score 2-5 | Detection and Visibility

Hope it clears.

[marcusbakker]

Thanks, that clears things up. Coincidentally, this was already on the roadmap for the next release :-). So, it's coming.

[rubinatorz]

hi @robotbond,

We just released a new version of DeTT&CT where we included different shades of colours in overlays. Hope you liked it.

Regards, Ruben

[robotbond]

Hi Ruben,

I downloaded the new version and found what was requested. Thanks for including different shades, it makes the reader job easy.

Regards, Nitin