Closed packet-rat closed 1 year ago
python3 misp_import.py -f -nb -d --obliterate
😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱
@@@ @@@ @@@ @@@@@@ @@@@@@@ @@@ @@@ @@@ @@@ @@@ @@@@@@@@ @@@
@@@ @@@ @@@ @@@@@@@@ @@@@@@@@ @@@@ @@@ @@@ @@@@ @@@ @@@@@@@@@ @@@
@@! @@! @@! @@! @@@ @@! @@@ @@!@!@@@ @@! @@!@!@@@ !@@ @@!
!@! !@! !@! !@! @!@ !@! @!@ !@!!@!@! !@! !@!!@!@! !@! !@
@!! !!@ @!@ @!@!@!@! @!@!!@! @!@ !!@! !!@ @!@ !!@! !@! @!@!@ @!@
!@! !!! !@! !!!@!!!! !!@!@! !@! !!! !!! !@! !!! !!! !!@!! !!!
!!: !!: !!: !!: !!! !!: :!! !!: !!! !!: !!: !!! :!! !!:
:!: :!: :!: :!: !:! :!: !:! :!: !:! :!: :!: !:! :!: !:: :!:
:::: :: ::: :: ::: :: ::: :: :: :: :: :: ::: :::: ::
:: : : : : : : : : : :: : : :: : :: :: : :::
😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱 😱
Obliterate is a destructive operation that will remove all CrowdStrike data
from your MISP instance. There is no going back once this process completes.
Are you sure you want to do this?
[Enter 'yes' to continue] ==> yes
_.-^^---....,,---;
_--/ `--_
< >)
| KA-BOOM! |
\._ _./
```--. . , ; .--'''
| | |
.-=|| | |=-.
`-=#$%&%$#=-'
| ; :|
_____.,-#%&$@%#&#~,._____
COMMAND ACCEPTED
[2023-03-31 19:16:11,848] INFO misp_tools MISP Import for CrowdStrike Threat Intelligence v0.6.8
[2023-03-31 19:16:11,849] INFO config CHECK CONFIG
[2023-03-31 19:16:11,850] DEBUG config client_id value redacted, check config file
[2023-03-31 19:16:11,850] DEBUG config client_secret value redacted, check config file
[2023-03-31 19:16:11,850] DEBUG config crowdstrike_url US1
[2023-03-31 19:16:11,850] DEBUG config api_request_max 5000
[2023-03-31 19:16:11,850] DEBUG config api_enable_ssl True
[2023-03-31 19:16:11,850] DEBUG config reports_timestamp_filename lastReportsUpdate.dat
[2023-03-31 19:16:11,850] DEBUG config indicators_timestamp_filename lastIndicatorsUpdate.dat
[2023-03-31 19:16:11,850] DEBUG config actors_timestamp_filename lastActorsUpdate.dat
[2023-03-31 19:16:11,850] DEBUG config init_reports_days_before 365
[2023-03-31 19:16:11,850] DEBUG config init_indicators_minutes_before 300
[2023-03-31 19:16:11,850] DEBUG config init_actors_days_before 365
[2023-03-31 19:16:11,850] DEBUG config reports_unique_tag CrowdStrike: REPORT
[2023-03-31 19:16:11,850] DEBUG config indicators_unique_tag CrowdStrike: INDICATOR
[2023-03-31 19:16:11,851] DEBUG config actors_unique_tag CrowdStrike: ADVERSARY
[2023-03-31 19:16:11,851] DEBUG config reports_tags att:source="Crowdstrike.Report"
[2023-03-31 19:16:11,851] DEBUG config indicators_tags att:source="Crowdstrike.Indicators"
[2023-03-31 19:16:11,851] DEBUG config actors_tags att:source="Crowdstrike.Actors"
[2023-03-31 19:16:11,851] DEBUG config unknown_mapping CrowdStrike:indicator:galaxy: UNATTRIBUTED
[2023-03-31 19:16:11,851] DEBUG config unattributed_title Unattributed indicators:
[2023-03-31 19:16:11,851] DEBUG config indicator_type_title Indicator Type:
[2023-03-31 19:16:11,851] DEBUG config malware_family_title Malware Family:
[2023-03-31 19:16:11,851] DEBUG config log_duplicates_as_sightings True
[2023-03-31 19:16:11,851] DEBUG config misp_url https://3samisp
[2023-03-31 19:16:11,851] DEBUG config misp_auth_key value redacted, check config file
[2023-03-31 19:16:11,851] DEBUG config crowdstrike_org_uuid ca4f4b5d-db04-4a5e-a6de-e60636dc01be
[2023-03-31 19:16:11,851] DEBUG config misp_enable_ssl False
[2023-03-31 19:16:11,851] WARNING config misp_enable_ssl SSL is disabled for MISP API requests
[2023-03-31 19:16:11,851] DEBUG config ind_attribute_batch_size 2500
[2023-03-31 19:16:11,851] DEBUG config event_save_memory_refresh_interval 180
[2023-03-31 19:16:11,851] DEBUG config max_threads 16
[2023-03-31 19:16:11,851] DEBUG config miss_track_file no_galaxy_mapping.log
[2023-03-31 19:16:11,851] DEBUG config galaxies_map_file galaxy.ini
[2023-03-31 19:16:11,851] DEBUG config tag_unknown_galaxy_maps True
[2023-03-31 19:16:11,851] DEBUG config taxonomic_kill-chain True
[2023-03-31 19:16:11,851] DEBUG config taxonomic_information-security-data-source True
[2023-03-31 19:16:11,851] DEBUG config taxonomic_type True
[2023-03-31 19:16:11,851] DEBUG config taxonomic_iep False
[2023-03-31 19:16:11,851] DEBUG config taxonomic_iep2 True
[2023-03-31 19:16:11,852] DEBUG config taxonomic_iep2_version False
[2023-03-31 19:16:11,852] DEBUG config taxonomic_tlp True
[2023-03-31 19:16:11,852] DEBUG config taxonomic_workflow True
[2023-03-31 19:16:12,753] INFO config No configuration errors found (1 warning)
[2023-03-31 19:16:12,753] INFO config
[2023-03-31 19:16:12,753] INFO config ____ _ _ ____ ____ _ _ ____ ___ ____ ____ ____ ____ ___
[2023-03-31 19:16:12,753] INFO config | |__| |___ | |_/ [__ |__] |__| [__ [__ |___ | \
[2023-03-31 19:16:12,753] INFO config |___ | | |___ |___ | \_ ___] | | | ___] ___] |___ |__/
[2023-03-31 19:16:12,753] INFO config
[2023-03-31 19:16:14,174] INFO processor/main BEGIN DELETE
[2023-03-31 19:16:14,174] INFO processor/main Start clean up of CrowdStrike Adversary (BAT) events from MISP.
[2023-03-31 19:16:14,322] INFO processor/main Completed deletion of CrowdStrike BAT adversaries within MISP in 0.14 seconds
[2023-03-31 19:16:14,322] INFO processor/main Start clean up of CrowdStrike Adversary (BEAR) events from MISP.
[2023-03-31 19:16:14,467] INFO processor/main Completed deletion of CrowdStrike BEAR adversaries within MISP in 0.14 seconds
[2023-03-31 19:16:14,467] INFO processor/main Start clean up of CrowdStrike Adversary (BUFFALO) events from MISP.
[2023-03-31 19:16:14,599] INFO processor/main Completed deletion of CrowdStrike BUFFALO adversaries within MISP in 0.13 seconds
[2023-03-31 19:16:14,599] INFO processor/main Start clean up of CrowdStrike Adversary (CHOLLIMA) events from MISP.
[2023-03-31 19:16:14,691] INFO processor/main Completed deletion of CrowdStrike CHOLLIMA adversaries within MISP in 0.09 seconds
[2023-03-31 19:16:14,691] INFO processor/main Start clean up of CrowdStrike Adversary (CRANE) events from MISP.
[2023-03-31 19:16:14,895] INFO processor/main Completed deletion of CrowdStrike CRANE adversaries within MISP in 0.20 seconds
[2023-03-31 19:16:14,895] INFO processor/main Start clean up of CrowdStrike Adversary (JACKAL) events from MISP.
[2023-03-31 19:16:14,997] INFO processor/main Completed deletion of CrowdStrike JACKAL adversaries within MISP in 0.10 seconds
[2023-03-31 19:16:14,997] INFO processor/main Start clean up of CrowdStrike Adversary (HAWK) events from MISP.
[2023-03-31 19:16:15,240] INFO processor/main Completed deletion of CrowdStrike HAWK adversaries within MISP in 0.24 seconds
[2023-03-31 19:16:15,240] INFO processor/main Start clean up of CrowdStrike Adversary (KITTEN) events from MISP.
[2023-03-31 19:16:15,328] INFO processor/main Completed deletion of CrowdStrike KITTEN adversaries within MISP in 0.08 seconds
[2023-03-31 19:16:15,328] INFO processor/main Start clean up of CrowdStrike Adversary (LEOPARD) events from MISP.
[2023-03-31 19:16:15,432] INFO processor/main Completed deletion of CrowdStrike LEOPARD adversaries within MISP in 0.10 seconds
[2023-03-31 19:16:15,432] INFO processor/main Start clean up of CrowdStrike Adversary (LYNX) events from MISP.
[2023-03-31 19:16:15,516] INFO processor/main Completed deletion of CrowdStrike LYNX adversaries within MISP in 0.08 seconds
[2023-03-31 19:16:15,516] INFO processor/main Start clean up of CrowdStrike Adversary (OCELOT) events from MISP.
[2023-03-31 19:16:15,592] INFO processor/main Completed deletion of CrowdStrike OCELOT adversaries within MISP in 0.07 seconds
[2023-03-31 19:16:15,592] INFO processor/main Start clean up of CrowdStrike Adversary (PANDA) events from MISP.
[2023-03-31 19:16:15,753] INFO processor/main Completed deletion of CrowdStrike PANDA adversaries within MISP in 0.16 seconds
[2023-03-31 19:16:15,753] INFO processor/main Start clean up of CrowdStrike Adversary (SPIDER) events from MISP.
[2023-03-31 19:16:15,872] INFO processor/main Completed deletion of CrowdStrike SPIDER adversaries within MISP in 0.11 seconds
[2023-03-31 19:16:15,873] INFO processor/main Start clean up of CrowdStrike Adversary (TIGER) events from MISP.
[2023-03-31 19:16:15,992] INFO processor/main Completed deletion of CrowdStrike TIGER adversaries within MISP in 0.11 seconds
[2023-03-31 19:16:15,992] INFO processor/main Start clean up of CrowdStrike Adversary (WOLF) events from MISP.
[2023-03-31 19:16:16,133] INFO processor/main Completed deletion of CrowdStrike WOLF adversaries within MISP in 0.14 seconds
[2023-03-31 19:16:16,133] INFO processor/main Start clean up of CrowdStrike CSA report events from MISP.
[2023-03-31 19:16:16,252] INFO processor/main Completed deletion of CrowdStrike CSA reports within MISP in 0.11 seconds
[2023-03-31 19:16:16,252] INFO processor/main Start clean up of CrowdStrike CSAR report events from MISP.
[2023-03-31 19:16:16,405] INFO processor/main Completed deletion of CrowdStrike CSAR reports within MISP in 0.15 seconds
[2023-03-31 19:16:16,405] INFO processor/main Start clean up of CrowdStrike CSIR report events from MISP.
[2023-03-31 19:16:16,538] INFO processor/main Completed deletion of CrowdStrike CSIR reports within MISP in 0.13 seconds
[2023-03-31 19:16:16,538] INFO processor/main Start clean up of CrowdStrike CSDR report events from MISP.
[2023-03-31 19:16:16,646] INFO processor/main Completed deletion of CrowdStrike CSDR reports within MISP in 0.10 seconds
[2023-03-31 19:16:16,646] INFO processor/main Start clean up of CrowdStrike CSIT report events from MISP.
[2023-03-31 19:16:16,736] INFO processor/main Completed deletion of CrowdStrike CSIT reports within MISP in 0.08 seconds
[2023-03-31 19:16:16,736] INFO processor/main Start clean up of CrowdStrike CSGT report events from MISP.
[2023-03-31 19:16:16,868] INFO processor/main Completed deletion of CrowdStrike CSGT reports within MISP in 0.13 seconds
[2023-03-31 19:16:16,868] INFO processor/main Start clean up of CrowdStrike CSIA report events from MISP.
[2023-03-31 19:16:16,944] INFO processor/main Completed deletion of CrowdStrike CSIA reports within MISP in 0.07 seconds
[2023-03-31 19:16:16,944] INFO processor/main Start clean up of CrowdStrike CSQR report events from MISP.
[2023-03-31 19:16:17,192] INFO processor/main Completed deletion of CrowdStrike CSQR reports within MISP in 0.24 seconds
[2023-03-31 19:16:17,192] INFO processor/main Start clean up of CrowdStrike CSMR report events from MISP.
[2023-03-31 19:16:17,452] INFO processor/main Completed deletion of CrowdStrike CSMR reports within MISP in 0.26 seconds
[2023-03-31 19:16:17,452] INFO processor/main Start clean up of CrowdStrike CSTA report events from MISP.
[2023-03-31 19:16:17,553] INFO processor/main Completed deletion of CrowdStrike CSTA reports within MISP in 0.10 seconds
[2023-03-31 19:16:17,553] INFO processor/main Start clean up of CrowdStrike CSWR report events from MISP.
[2023-03-31 19:16:17,638] INFO processor/main Completed deletion of CrowdStrike CSWR reports within MISP in 0.08 seconds
[2023-03-31 19:16:17,639] INFO processor/main Start clean up of CrowdStrike TEST report events from MISP.
[2023-03-31 19:16:17,713] INFO processor/main Completed deletion of CrowdStrike TEST reports within MISP in 0.07 seconds
[2023-03-31 19:16:17,713] INFO processor/main Start clean up of CrowdStrike HASH_MD5 indicator events from MISP.
[2023-03-31 19:16:17,811] INFO processor/main Start clean up of CrowdStrike HASH_SHA256 indicator events from MISP.
[2023-03-31 19:16:17,897] INFO processor/main Start clean up of CrowdStrike HASH_SHA1 indicator events from MISP.
[2023-03-31 19:16:17,982] INFO processor/main Start clean up of CrowdStrike HASH_IMPHASH indicator events from MISP.
[2023-03-31 19:16:18,268] INFO processor/main Start clean up of CrowdStrike FILE_NAME indicator events from MISP.
[2023-03-31 19:16:18,355] INFO processor/main Start clean up of CrowdStrike FILE_PATH indicator events from MISP.
[2023-03-31 19:16:18,426] INFO processor/main Start clean up of CrowdStrike URL indicator events from MISP.
[2023-03-31 19:16:18,536] INFO processor/main Start clean up of CrowdStrike MUTEX_NAME indicator events from MISP.
[2023-03-31 19:16:18,618] INFO processor/main Start clean up of CrowdStrike BITCOIN_ADDRESS indicator events from MISP.
[2023-03-31 19:16:18,718] INFO processor/main Start clean up of CrowdStrike COIN_ADDRESS indicator events from MISP.
[2023-03-31 19:16:18,795] INFO processor/main Start clean up of CrowdStrike EMAIL_ADDRESS indicator events from MISP.
[2023-03-31 19:16:18,871] INFO processor/main Start clean up of CrowdStrike EMAIL_SUBJECT indicator events from MISP.
[2023-03-31 19:16:18,949] INFO processor/main Start clean up of CrowdStrike REGISTRY indicator events from MISP.
[2023-03-31 19:16:19,156] INFO processor/main Start clean up of CrowdStrike DEVICE_NAME indicator events from MISP.
[2023-03-31 19:16:19,336] INFO processor/main Start clean up of CrowdStrike DOMAIN indicator events from MISP.
[2023-03-31 19:16:19,466] INFO processor/main Start clean up of CrowdStrike CAMPAIGN_ID indicator events from MISP.
[2023-03-31 19:16:19,640] INFO processor/main Start clean up of CrowdStrike IP_ADDRESS indicator events from MISP.
[2023-03-31 19:16:19,724] INFO processor/main Start clean up of CrowdStrike SERVICE_NAME indicator events from MISP.
[2023-03-31 19:16:19,831] INFO processor/main Start clean up of CrowdStrike USER_AGENT indicator events from MISP.
[2023-03-31 19:16:19,928] INFO processor/main Start clean up of CrowdStrike PORT indicator events from MISP.
[2023-03-31 19:16:19,990] INFO processor/main Start clean up of CrowdStrike PASSWORD indicator events from MISP.
[2023-03-31 19:16:20,247] INFO processor/main Start clean up of CrowdStrike USERNAME indicator events from MISP.
[2023-03-31 19:16:20,377] INFO processor/main Start clean up of CrowdStrike X509_SERIAL indicator events from MISP.
[2023-03-31 19:16:20,509] INFO processor/main Start clean up of CrowdStrike X509_SUBJECT indicator events from MISP.
[2023-03-31 19:16:20,692] INFO processor/main Start clean up of CrowdStrike MD5 hashes indicator type events from MISP.
[2023-03-31 19:16:20,757] INFO processor/main Start clean up of CrowdStrike SHA256 hashes indicator type events from MISP.
[2023-03-31 19:16:20,849] INFO processor/main Start clean up of CrowdStrike SHA1 hashes indicator type events from MISP.
[2023-03-31 19:16:20,939] INFO processor/main Start clean up of CrowdStrike IMP hashes indicator type events from MISP.
[2023-03-31 19:16:21,133] INFO processor/main Start clean up of CrowdStrike File names indicator type events from MISP.
[2023-03-31 19:16:21,235] INFO processor/main Start clean up of CrowdStrike File directory paths indicator type events from MISP.
[2023-03-31 19:16:21,374] INFO processor/main Start clean up of CrowdStrike Web addresses indicator type events from MISP.
[2023-03-31 19:16:21,498] INFO processor/main Start clean up of CrowdStrike Mutexes indicator type events from MISP.
[2023-03-31 19:16:21,598] INFO processor/main Start clean up of CrowdStrike BTC addresses indicator type events from MISP.
[2023-03-31 19:16:21,715] INFO processor/main Start clean up of CrowdStrike BIC addresses indicator type events from MISP.
[2023-03-31 19:16:21,805] INFO processor/main Start clean up of CrowdStrike Email addresses indicator type events from MISP.
[2023-03-31 19:16:21,943] INFO processor/main Start clean up of CrowdStrike Email subjects indicator type events from MISP.
[2023-03-31 19:16:22,116] INFO processor/main Start clean up of CrowdStrike Registry key locations indicator type events from MISP.
[2023-03-31 19:16:22,210] INFO processor/main Start clean up of CrowdStrike Device host names indicator type events from MISP.
[2023-03-31 19:16:22,292] INFO processor/main Start clean up of CrowdStrike Web domains indicator type events from MISP.
[2023-03-31 19:16:22,355] INFO processor/main Start clean up of CrowdStrike Campaign IDs indicator type events from MISP.
[2023-03-31 19:16:22,445] INFO processor/main Start clean up of CrowdStrike IP addresses indicator type events from MISP.
[2023-03-31 19:16:22,581] INFO processor/main Start clean up of CrowdStrike Service names indicator type events from MISP.
[2023-03-31 19:16:22,726] INFO processor/main Start clean up of CrowdStrike User-Agent strings indicator type events from MISP.
[2023-03-31 19:16:22,896] INFO processor/main Start clean up of CrowdStrike TCP ports indicator type events from MISP.
[2023-03-31 19:16:22,964] INFO processor/main Start clean up of CrowdStrike Password credentials indicator type events from MISP.
[2023-03-31 19:16:23,216] INFO processor/main Start clean up of CrowdStrike Credential user names indicator type events from MISP.
[2023-03-31 19:16:23,360] INFO processor/main Start clean up of CrowdStrike Certificate serial numbers indicator type events from MISP.
[2023-03-31 19:16:23,465] INFO processor/main Start clean up of CrowdStrike Certificate subjects indicator type events from MISP.
[2023-03-31 19:16:23,682] INFO processor/main Start clean up of CrowdStrike malware family indicator events from MISP.
[2023-03-31 19:16:23,806] INFO processor/main Completed deletion of CrowdStrike indicators within MISP in 6.09 seconds
[2023-03-31 19:16:23,806] INFO processor/main Finished cleaning up CrowdStrike related events from MISP, 0 events deleted.
[2023-03-31 19:16:23,806] INFO processor/main BEGIN DELETE
[2023-03-31 19:16:23,807] INFO processor/main Retrieving list of tags to remove from MISP instance (may take several minutes).
[2023-03-31 19:16:24,194] INFO processor/main Finished cleaning up CrowdStrike related tags from MISP, 0 tags deleted.
[2023-03-31 19:16:24,194] INFO misp_tools FINISHED
I believe I may have found the Root Cause for this issue:
I'd deleted the CrowdStrike Tags as part of another set of Tag related Issues.
I suspect you may be totally relying on Tags to identify the Obliterate target list. If so, a more resilient methodology would include criteria like Organization. We'd still need some parameter driven tag specification. In our case our AT&T specific Tags we apply would be used for the deletions that focus on a given category (i.e., Actors, Reports, Indicators).
Let's validate the assumption that bulk deletions are currently based solely on Tags and take it from there?
After doing a manual BulkDeletion, FullMonty, and then an Obliterate - the following artifacts remained:
Interesting. The two events above do not have the CrowdStrike tag used to identify records for deletion. I've seen this before, but only when I crashed / stopped import in a non-graceful manner.
If you cleared the tags, then I'm sure that's the cause. I've tried adding in org ID to the filter used to select deleted records, and that doesn't seem to work using pymisp. That was a couple versions back, so I can retry this in a follow up revision.
Closing this one for now, as I think we've found it. Feel free to reopen if you find another example.
Crowdstrike Artifacts (i.e., Reports, Actors) are not deleted after performing Obliterate and Clean functions functions.