search

CrowdStrike / MISP-tools

Import CrowdStrike Threat Intelligence into your instance of MISP
MIT License
41 stars 10 forks source link

--clean_xxx does not remove any Events (Reports, Actors, Indicators) #33

Closed packet-rat closed 2 years ago

packet-rat commented 2 years ago

Function --clean_reports does not remove any reports

(1) Loaded 1,000 of Reports (2) Executed --clean_reports (3) No Reports deleted.

(venv) [rx118r@md2nj01di:~/src/crowdstrike/220812/MISP-tools-main]$ python3 misp_import.py --reports

[2022-08-13 16:06:28,448] (INFO) 
'##::::'##:'####::'######::'########:::::'####:'##::::'##:'########:::'#######::'########::'########:
 ###::'###:. ##::'##... ##: ##.... ##::::. ##:: ###::'###: ##.... ##:'##.... ##: ##.... ##:... ##..::
 ####'####:: ##:: ##:::..:: ##:::: ##::::: ##:: ####'####: ##:::: ##: ##:::: ##: ##:::: ##:::: ##::::
 ## ### ##:: ##::. ######:: ########:::::: ##:: ## ### ##: ########:: ##:::: ##: ########::::: ##::::
 ##. #: ##:: ##:::..... ##: ##.....::::::: ##:: ##. #: ##: ##.....::: ##:::: ##: ##.. ##:::::: ##::::
 ##:.:: ##:: ##::'##::: ##: ##:::::::::::: ##:: ##:.:: ##: ##:::::::: ##:::: ##: ##::. ##::::: ##::::
 ##:::: ##:'####:. ######:: ##:::::::::::'####: ##:::: ##: ##::::::::. #######:: ##:::. ##:::: ##::::
..:::::..::....:::......:::..::::::::::::....::..:::::..::..::::::::::.......:::..:::::..:::::..:::::
                      _____
                       /  '
                    ,-/-,__ __
                   (_/  (_)/ (_
                               _______                        __ _______ __        __ __
                              |   _   .----.-----.--.--.--.--|  |   _   |  |_.----|__|  |--.-----.
                              |.  1___|   _|  _  |  |  |  |  _  |   1___|   _|   _|  |    <|  -__|
                              |.  |___|__| |_____|________|_____|____   |____|__| |__|__|__|_____|
                              |:  1   |                         |:  1   |
                              |::.. . |                         |::.. . |    Threat Intelligence
                              `-------'                         `-------'

[2022-08-13 16:06:29,202] (INFO) 
 ______  _______  ______ _____ __   _      _____ _______  _____   _____   ______ _______
 |_____] |______ |  ____   |   | \  |        |   |  |  | |_____] |     | |_____/    |
 |_____] |______ |_____| __|__ |  \_|      __|__ |  |  | |       |_____| |    \_    |

[2022-08-13 16:06:29,202] (INFO) 
 ____     ___  ____    ___   ____  ______  _____
|    \   /  _]|    \  /   \ |    \|      T/ ___/
|  D  ) /  [_ |  o  )Y     Y|  D  )      (   \_
|    / Y    _]|   _/ |  O  ||    /l_j  l_j\__  T
|    \ |   [_ |  |   |     ||    \  |  |  /  \ |
|  .  Y|     T|  |   l     !|  .  Y |  |  \    |
l__j\_jl_____jl__j    \___/ l__j\_j l__j   \___j

[2022-08-13 16:06:29,202] (INFO) Start getting reports from Crowdstrike Intel API and pushing them as events in MISP (past 365 days).
[2022-08-13 16:07:19,070] (INFO) Got 10728 reports from the Crowdstrike Intel API.
[2022-08-13 16:08:51,412] (INFO) Retrieved extended report details for 9884 reports
[2022-08-13 16:09:22,620] (INFO) 99355 related indicators found
[2022-08-13 16:09:22,906] (INFO) CSIT-20205 RedLine Stealer, Bond Loader, and the Author "REDGlade" report created.
[2022-08-13 16:09:23,021] (INFO) CSA-17098 Emotet Banking Trojan Distribution Continues Via Delivery Notification Spam report created.
[2022-08-13 16:09:23,038] (INFO) CSA-210621 Russian Actors Advertise Big Game Hunting Ransomware on African eCrime Channels; Claims of REvil Partnership Likely False report created.
[2022-08-13 16:09:23,060] (INFO) CSA-16357 Bitcoin Investments and Price Volatility Highlight China's Growing Influence on Top Cryptocurrency report created.
[2022-08-13 16:09:23,061] (INFO) CSA-16407 New Version of Petya Ransomware Available with Dual-layer Access Prevention Included; U.S. and European Countries Likely to be Targeted report created.
[2022-08-13 16:09:23,175] (INFO) CSWR-16010 GTAC Weekly Wrap-Up: Week of 3/12/16 report created.
[2022-08-13 16:09:23,231] (INFO) CSWR-16021 GTAC Weekly Wrap-Up: Week of 5/28/16 report created.
[2022-08-13 16:09:23,394] (INFO) CSA-16349 United Cyber Caliphate Publishes Database of Saudi Citizens Indicating the Group Remains Active Despite Recent Arrests report created.
[2022-08-13 16:09:23,556] (INFO) CSA-16342 Malicious Operation Demonstrates Sophisticated Knowledge of Cloud Storage Environment; Suspected Ties to FANCY BEAR report created.
[2022-08-13 16:09:23,594] (INFO) CSIT-17127 XDATA Ransomware Deployed Via M.E.Doc Update Mechanism report created.
[2022-08-13 16:09:23,616] (INFO) CSA-16348 Asian Football Confederation Website Defaced by Iranian Actor; Probable Links to Iranian Revolutionary Posturing report created.
[2022-08-13 16:09:23,769] (INFO) CSA-16344 Member Seeks Distribution Partners for Russian-Sourced Ransomware on Brazilian Underground eCrime Forum report created.
[2022-08-13 16:09:23,786] (INFO) CSA-16360 Quant Loader Hits Underground Forums; Increases Competition in Loader Market report created.

<SNIP>

(venv) [rx118r@md2nj01di:~/src/crowdstrike/220812/MISP-tools-main]$ python3 misp_import.py --clean_reports


[2022-08-13 16:14:39,658] (INFO) 
'##::::'##:'####::'######::'########:::::'####:'##::::'##:'########:::'#######::'########::'########:
 ###::'###:. ##::'##... ##: ##.... ##::::. ##:: ###::'###: ##.... ##:'##.... ##: ##.... ##:... ##..::
 ####'####:: ##:: ##:::..:: ##:::: ##::::: ##:: ####'####: ##:::: ##: ##:::: ##: ##:::: ##:::: ##::::
 ## ### ##:: ##::. ######:: ########:::::: ##:: ## ### ##: ########:: ##:::: ##: ########::::: ##::::
 ##. #: ##:: ##:::..... ##: ##.....::::::: ##:: ##. #: ##: ##.....::: ##:::: ##: ##.. ##:::::: ##::::
 ##:.:: ##:: ##::'##::: ##: ##:::::::::::: ##:: ##:.:: ##: ##:::::::: ##:::: ##: ##::. ##::::: ##::::
 ##:::: ##:'####:. ######:: ##:::::::::::'####: ##:::: ##: ##::::::::. #######:: ##:::. ##:::: ##::::
..:::::..::....:::......:::..::::::::::::....::..:::::..::..::::::::::.......:::..:::::..:::::..:::::
                      _____
                       /  '
                    ,-/-,__ __
                   (_/  (_)/ (_
                               _______                        __ _______ __        __ __
                              |   _   .----.-----.--.--.--.--|  |   _   |  |_.----|__|  |--.-----.
                              |.  1___|   _|  _  |  |  |  |  _  |   1___|   _|   _|  |    <|  -__|
                              |.  |___|__| |_____|________|_____|____   |____|__| |__|__|__|_____|
                              |:  1   |                         |:  1   |
                              |::.. . |                         |::.. . |    Threat Intelligence
                              `-------'                         `-------'

[2022-08-13 16:14:40,599] (INFO) 
 ______  _______  ______ _____ __   _      ______  _______        _______ _______ _______
 |_____] |______ |  ____   |   | \  |      |     \ |______ |      |______    |    |______
 |_____] |______ |_____| __|__ |  \_|      |_____/ |______ |_____ |______    |    |______

[2022-08-13 16:14:40,600] (INFO) Start clean up CrowdStrike related events from MISP.
[2022-08-13 16:14:40,632] (INFO) Finished cleaning up CrowdStrike related events from MISP, 0 events deleted.
[2022-08-13 16:14:40,632] (INFO) 
 _______ _____ __   _ _____ _______ _     _ _______ ______
 |______   |   | \  |   |   |______ |_____| |______ |     \
 |       __|__ |  \_| __|__ ______| |     | |______ |_____/
packet-rat commented 2 years ago

Same for --clean_indicators, --clean_actors


(venv) [rx118r@md2nj01di:~/src/crowdstrike/220812/MISP-tools-main]$ python3 misp_import.py --clean_indicators
[2022-08-13 16:21:51,210] (INFO) 
'##::::'##:'####::'######::'########:::::'####:'##::::'##:'########:::'#######::'########::'########:
 ###::'###:. ##::'##... ##: ##.... ##::::. ##:: ###::'###: ##.... ##:'##.... ##: ##.... ##:... ##..::
 ####'####:: ##:: ##:::..:: ##:::: ##::::: ##:: ####'####: ##:::: ##: ##:::: ##: ##:::: ##:::: ##::::
 ## ### ##:: ##::. ######:: ########:::::: ##:: ## ### ##: ########:: ##:::: ##: ########::::: ##::::
 ##. #: ##:: ##:::..... ##: ##.....::::::: ##:: ##. #: ##: ##.....::: ##:::: ##: ##.. ##:::::: ##::::
 ##:.:: ##:: ##::'##::: ##: ##:::::::::::: ##:: ##:.:: ##: ##:::::::: ##:::: ##: ##::. ##::::: ##::::
 ##:::: ##:'####:. ######:: ##:::::::::::'####: ##:::: ##: ##::::::::. #######:: ##:::. ##:::: ##::::
..:::::..::....:::......:::..::::::::::::....::..:::::..::..::::::::::.......:::..:::::..:::::..:::::
                      _____
                       /  '
                    ,-/-,__ __
                   (_/  (_)/ (_
                               _______                        __ _______ __        __ __
                              |   _   .----.-----.--.--.--.--|  |   _   |  |_.----|__|  |--.-----.
                              |.  1___|   _|  _  |  |  |  |  _  |   1___|   _|   _|  |    <|  -__|
                              |.  |___|__| |_____|________|_____|____   |____|__| |__|__|__|_____|
                              |:  1   |                         |:  1   |
                              |::.. . |                         |::.. . |    Threat Intelligence
                              `-------'                         `-------'

[2022-08-13 16:21:51,863] (INFO) 
 ______  _______  ______ _____ __   _      ______  _______        _______ _______ _______
 |_____] |______ |  ____   |   | \  |      |     \ |______ |      |______    |    |______
 |_____] |______ |_____| __|__ |  \_|      |_____/ |______ |_____ |______    |    |______

[2022-08-13 16:21:51,863] (INFO) Start clean up CrowdStrike related events from MISP.
[2022-08-13 16:21:51,895] (INFO) Finished cleaning up CrowdStrike related events from MISP, 0 events deleted.
[2022-08-13 16:21:51,896] (INFO) 
 _______ _____ __   _ _____ _______ _     _ _______ ______
 |______   |   | \  |   |   |______ |_____| |______ |     \
 |       __|__ |  \_| __|__ ______| |     | |______ |_____/

(venv) [rx118r@md2nj01di:~/src/crowdstrike/220812/MISP-tools-main]$ python3 misp_import.py --clean_actors
[2022-08-13 16:22:10,716] (INFO) 
'##::::'##:'####::'######::'########:::::'####:'##::::'##:'########:::'#######::'########::'########:
 ###::'###:. ##::'##... ##: ##.... ##::::. ##:: ###::'###: ##.... ##:'##.... ##: ##.... ##:... ##..::
 ####'####:: ##:: ##:::..:: ##:::: ##::::: ##:: ####'####: ##:::: ##: ##:::: ##: ##:::: ##:::: ##::::
 ## ### ##:: ##::. ######:: ########:::::: ##:: ## ### ##: ########:: ##:::: ##: ########::::: ##::::
 ##. #: ##:: ##:::..... ##: ##.....::::::: ##:: ##. #: ##: ##.....::: ##:::: ##: ##.. ##:::::: ##::::
 ##:.:: ##:: ##::'##::: ##: ##:::::::::::: ##:: ##:.:: ##: ##:::::::: ##:::: ##: ##::. ##::::: ##::::
 ##:::: ##:'####:. ######:: ##:::::::::::'####: ##:::: ##: ##::::::::. #######:: ##:::. ##:::: ##::::
..:::::..::....:::......:::..::::::::::::....::..:::::..::..::::::::::.......:::..:::::..:::::..:::::
                      _____
                       /  '
                    ,-/-,__ __
                   (_/  (_)/ (_
                               _______                        __ _______ __        __ __
                              |   _   .----.-----.--.--.--.--|  |   _   |  |_.----|__|  |--.-----.
                              |.  1___|   _|  _  |  |  |  |  _  |   1___|   _|   _|  |    <|  -__|
                              |.  |___|__| |_____|________|_____|____   |____|__| |__|__|__|_____|
                              |:  1   |                         |:  1   |
                              |::.. . |                         |::.. . |    Threat Intelligence
                              `-------'                         `-------'

[2022-08-13 16:22:11,696] (INFO) 
 ______  _______  ______ _____ __   _      ______  _______        _______ _______ _______
 |_____] |______ |  ____   |   | \  |      |     \ |______ |      |______    |    |______
 |_____] |______ |_____| __|__ |  \_|      |_____/ |______ |_____ |______    |    |______

[2022-08-13 16:22:11,696] (INFO) Start clean up CrowdStrike related events from MISP.
[2022-08-13 16:22:11,729] (INFO) Finished cleaning up CrowdStrike related events from MISP, 0 events deleted.
[2022-08-13 16:22:11,729] (INFO) 
 _______ _____ __   _ _____ _______ _     _ _______ ______
 |______   |   | \  |   |   |______ |_____| |______ |     \
 |       __|__ |  \_| __|__ ______| |     | |______ |_____/

(venv) [rx118r@md2nj01di:~/src/crowdstrike/220812/MISP-tools-main]$
jshcodes commented 2 years ago

Let me merge the version I'm working on right now, it has refined deletion logic. (Since local tags are used, you may have to manually delete any events without the correct tagging.)

Stand by.

jshcodes commented 2 years ago

Ok, grab the latest version and try an import / delete cycle again. You should see better tagging in all three event type, that should make searching easier. You should also be able to remove them by type (clean_adversary, clean_indicator, etc.).

Deletes are broken up a bit to try and reduce the query result sizes coming back from the MISP server.

There is also a --clean_tags command line argument that will remove all CrowdStrike local tags from your instance. After you've cleaned up your instance and removed any dangling events, run this (only) and remove all the CrowdStrike tags. Also make sure to remove any .dat files from your directory.

Do not run --clean_tags until you have removed all CrowdStrike events from your MISP instance.

You should see the version number reflected in the banner now. (Currently v0.6.1.)