search

swimlane / pyattck

A Python package to interact with the Mitre ATT&CK Framework
MIT License
468 stars 87 forks source link

Downloading fresh JSON (update) loops unnecessarily? #74

Closed xakepnz closed 3 years ago

xakepnz commented 3 years ago

I want to utilise local config as much as I can, to help with speed. When you had the .update() method I would utilise that. Now with the latest update, I need to build my own "update function" to overwrite the existing config, and any normal call, would just use the local json.

I've tried updating as below (fresh virtualenv) using pyattck-4.0.3 on Python 3.9.2 from pip:

>>> from pyattck import Attck
>>> import logging
>>>
>>> logging.basicConfig(format = '%(asctime)s - %(levelname)s - %(message)s',level = 'DEBUG')
>>>
>>> attack_update = Attck(
...     use_config=False,
...     save_config=True,
...     data_path='/home/xakep/derp/'
... )
>>>
>>> if attack_update:
...     print('yes')
...
yes
>>>
>>> if attack_update.enterprise:
...     print('yes')
...

Only now begins the download of the JSON files, and appears to download the same files 6 times:

2021-07-21 12:19:24,522 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:26,339 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/enterprise-attack/enterprise-attack.json HTTP/1.1" 200 2910652
2021-07-21 12:19:27,191 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:27,419 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/pre-attack/pre-attack.json HTTP/1.1" 200 85794
2021-07-21 12:19:27,464 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:27,628 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/mobile-attack/mobile-attack.json HTTP/1.1" 200 228573
2021-07-21 12:19:27,748 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:27,990 - DEBUG - https://raw.githubusercontent.com:443 "GET /center-for-threat-informed-defense/attack-control-framework-mappings/master/frameworks/nist800-53-r4/stix/nist800-53-r4-controls.json HTTP/1.1" 200 331153
2021-07-21 12:19:28,117 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:28,350 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/generated_attck_data.json?raw=True HTTP/1.1" 302 138
2021-07-21 12:19:28,564 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/generated_attck_data.json HTTP/1.1" 302 157
2021-07-21 12:19:28,567 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:29,957 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/generated_attck_data.json HTTP/1.1" 200 31396583
2021-07-21 12:19:31,359 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:31,577 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/attck_to_nist_controls.json?raw=True HTTP/1.1" 302 140
2021-07-21 12:19:31,778 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/attck_to_nist_controls.json HTTP/1.1" 302 159
2021-07-21 12:19:31,781 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:32,390 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/attck_to_nist_controls.json HTTP/1.1" 200 9105826
2021-07-21 12:19:32,704 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:32,734 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/generated_attck_data.json?raw=True HTTP/1.1" 302 138
2021-07-21 12:19:32,743 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/generated_attck_data.json HTTP/1.1" 302 157
2021-07-21 12:19:32,745 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:32,759 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/generated_attck_data.json HTTP/1.1" 200 31396583
2021-07-21 12:19:33,259 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:33,277 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/enterprise-attack/enterprise-attack.json HTTP/1.1" 200 2910652
2021-07-21 12:19:34,169 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:34,185 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/pre-attack/pre-attack.json HTTP/1.1" 200 85794
2021-07-21 12:19:34,229 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:34,243 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/mobile-attack/mobile-attack.json HTTP/1.1" 200 228573
2021-07-21 12:19:34,359 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:34,375 - DEBUG - https://raw.githubusercontent.com:443 "GET /center-for-threat-informed-defense/attack-control-framework-mappings/master/frameworks/nist800-53-r4/stix/nist800-53-r4-controls.json HTTP/1.1" 200 331153
2021-07-21 12:19:34,496 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:34,535 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/generated_attck_data.json?raw=True HTTP/1.1" 302 138
2021-07-21 12:19:34,549 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/generated_attck_data.json HTTP/1.1" 302 157
2021-07-21 12:19:34,551 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:34,575 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/generated_attck_data.json HTTP/1.1" 200 31396583
2021-07-21 12:19:36,646 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:36,673 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/attck_to_nist_controls.json?raw=True HTTP/1.1" 302 140
2021-07-21 12:19:36,679 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/attck_to_nist_controls.json HTTP/1.1" 302 159
2021-07-21 12:19:36,682 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:36,696 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/attck_to_nist_controls.json HTTP/1.1" 200 9105826
2021-07-21 12:19:37,041 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:37,057 - DEBUG - https://raw.githubusercontent.com:443 "GET /center-for-threat-informed-defense/attack-control-framework-mappings/master/frameworks/nist800-53-r4/stix/nist800-53-r4-controls.json HTTP/1.1" 200 331153
2021-07-21 12:19:37,090 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:37,114 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/enterprise-attack/enterprise-attack.json HTTP/1.1" 200 2910652
2021-07-21 12:19:38,014 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:38,029 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/pre-attack/pre-attack.json HTTP/1.1" 200 85794
2021-07-21 12:19:38,071 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:38,086 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/mobile-attack/mobile-attack.json HTTP/1.1" 200 228573
2021-07-21 12:19:38,173 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:38,188 - DEBUG - https://raw.githubusercontent.com:443 "GET /center-for-threat-informed-defense/attack-control-framework-mappings/master/frameworks/nist800-53-r4/stix/nist800-53-r4-controls.json HTTP/1.1" 200 331153
2021-07-21 12:19:38,293 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:38,321 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/generated_attck_data.json?raw=True HTTP/1.1" 302 138
2021-07-21 12:19:38,330 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/generated_attck_data.json HTTP/1.1" 302 157
2021-07-21 12:19:38,333 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:38,348 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/generated_attck_data.json HTTP/1.1" 200 31396583
2021-07-21 12:19:40,455 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:40,489 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/attck_to_nist_controls.json?raw=True HTTP/1.1" 302 140
2021-07-21 12:19:40,500 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/attck_to_nist_controls.json HTTP/1.1" 302 159
2021-07-21 12:19:40,503 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:40,517 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/attck_to_nist_controls.json HTTP/1.1" 200 9105826
2021-07-21 12:19:40,836 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:40,866 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/attck_to_nist_controls.json?raw=True HTTP/1.1" 302 140
2021-07-21 12:19:40,874 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/attck_to_nist_controls.json HTTP/1.1" 302 159
2021-07-21 12:19:40,877 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:40,890 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/attck_to_nist_controls.json HTTP/1.1" 200 9105826
2021-07-21 12:19:41,157 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:41,175 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/enterprise-attack/enterprise-attack.json HTTP/1.1" 200 2910652
2021-07-21 12:19:42,030 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:42,046 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/pre-attack/pre-attack.json HTTP/1.1" 200 85794
2021-07-21 12:19:42,154 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:42,179 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/mobile-attack/mobile-attack.json HTTP/1.1" 200 228573
2021-07-21 12:19:42,283 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:42,299 - DEBUG - https://raw.githubusercontent.com:443 "GET /center-for-threat-informed-defense/attack-control-framework-mappings/master/frameworks/nist800-53-r4/stix/nist800-53-r4-controls.json HTTP/1.1" 200 331153
2021-07-21 12:19:42,419 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:42,450 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/generated_attck_data.json?raw=True HTTP/1.1" 302 138
2021-07-21 12:19:42,460 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/generated_attck_data.json HTTP/1.1" 302 157
2021-07-21 12:19:42,463 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:42,479 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/generated_attck_data.json HTTP/1.1" 200 31396583
2021-07-21 12:19:44,938 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:44,963 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/attck_to_nist_controls.json?raw=True HTTP/1.1" 302 140
2021-07-21 12:19:44,970 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/attck_to_nist_controls.json HTTP/1.1" 302 159
2021-07-21 12:19:44,974 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:44,995 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/attck_to_nist_controls.json HTTP/1.1" 200 9105826
2021-07-21 12:19:45,334 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:45,350 - DEBUG - https://raw.githubusercontent.com:443 "GET /center-for-threat-informed-defense/attack-control-framework-mappings/master/frameworks/nist800-53-r4/stix/nist800-53-r4-controls.json HTTP/1.1" 200 331153
2021-07-21 12:19:45,421 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:45,437 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/enterprise-attack/enterprise-attack.json HTTP/1.1" 200 2910652
2021-07-21 12:19:46,304 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:46,318 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/pre-attack/pre-attack.json HTTP/1.1" 200 85794
2021-07-21 12:19:46,365 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:46,379 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/mobile-attack/mobile-attack.json HTTP/1.1" 200 228573
2021-07-21 12:19:46,483 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:46,499 - DEBUG - https://raw.githubusercontent.com:443 "GET /center-for-threat-informed-defense/attack-control-framework-mappings/master/frameworks/nist800-53-r4/stix/nist800-53-r4-controls.json HTTP/1.1" 200 331153
2021-07-21 12:19:46,643 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:46,675 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/generated_attck_data.json?raw=True HTTP/1.1" 302 138
2021-07-21 12:19:46,687 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/generated_attck_data.json HTTP/1.1" 302 157
2021-07-21 12:19:46,690 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:46,704 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/generated_attck_data.json HTTP/1.1" 200 31396583
2021-07-21 12:19:48,808 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:48,840 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/attck_to_nist_controls.json?raw=True HTTP/1.1" 302 140
2021-07-21 12:19:48,847 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/attck_to_nist_controls.json HTTP/1.1" 302 159
2021-07-21 12:19:48,850 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:48,869 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/attck_to_nist_controls.json HTTP/1.1" 200 9105826
2021-07-21 12:19:49,200 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:49,230 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/enterprise-attack/enterprise-attack.json HTTP/1.1" 200 2910652
yes
>>>

I believe the issue may be from the iteration over the JSON file names, and instead of a single-download it's downloading multiple? Am I wrong? And how do I try to implement an "update" func/method to only overwrite the json that exists using the new version 4.

Ref: https://github.com/swimlane/pyattck/blob/719e3582810c1728f6e7a6e020ec9249526c4cda/pyattck/configuration.py#L50

        for json_data in ['enterprise_attck_json', 'pre_attck_json', 
                          'mobile_attck_json', 'nist_controls_json', 
                          'generated_attck_json', 'generated_nist_json']:
            if cls._check_if_url(getattr(cls, json_data)):
                path = os.path.join(cls.data_path, "{json_data}.json".format(json_data=json_data))
                data = cls.__download_url_data(getattr(cls, json_data))
                cls.__write_to_disk(path, data)

PS: Great framework!

Thanks

MSAdministrator commented 3 years ago

@xakepnz interesting and thanks for reporting this! I’ll take a look at why it’s downloading multiple times - it shouldn’t be.

As far as the update functionality, I can definitely add that back in. Please create another issue for that and I’ll add it in the next release.

MSAdministrator commented 3 years ago

@xakepnz I believe I have a fix for this and should be pushing a change soon.