search

CrowdStrike / MISP-tools

Import CrowdStrike Threat Intelligence into your instance of MISP
MIT License
39 stars 10 forks source link

Fatal Error on Indicator Import: KeyError: 'PERSONA_NAME' #112

Closed packet-rat closed 1 year ago

packet-rat commented 1 year ago

python3 misp_import.py -f -nb -d -i

[2023-04-05 19:21:01,715] INFO     misp_tools    MISP Import for CrowdStrike Threat Intelligence v0.6.8
[2023-04-05 19:21:01,715] INFO     config  CHECK CONFIG
[2023-04-05 19:21:01,716] DEBUG    config  client_id                                   value redacted, check config file
[2023-04-05 19:21:01,716] DEBUG    config  client_secret                               value redacted, check config file
[2023-04-05 19:21:01,716] DEBUG    config  crowdstrike_url                             US1
[2023-04-05 19:21:01,716] DEBUG    config  api_request_max                             5000
[2023-04-05 19:21:01,716] DEBUG    config  api_enable_ssl                              True
[2023-04-05 19:21:01,716] DEBUG    config  reports_timestamp_filename                  lastReportsUpdate.dat
[2023-04-05 19:21:01,717] DEBUG    config  indicators_timestamp_filename               lastIndicatorsUpdate.dat
[2023-04-05 19:21:01,717] DEBUG    config  actors_timestamp_filename                   lastActorsUpdate.dat
[2023-04-05 19:21:01,717] DEBUG    config  init_reports_days_before                    365
[2023-04-05 19:21:01,717] DEBUG    config  init_indicators_minutes_before              20220
[2023-04-05 19:21:01,717] DEBUG    config  init_actors_days_before                     730
[2023-04-05 19:21:01,717] DEBUG    config  reports_unique_tag                          CrowdStrike: REPORT
[2023-04-05 19:21:01,717] DEBUG    config  indicators_unique_tag                       CrowdStrike: INDICATOR
[2023-04-05 19:21:01,717] DEBUG    config  actors_unique_tag                           CrowdStrike: ACTOR
[2023-04-05 19:21:01,717] DEBUG    config  reports_tags                                att:source="Crowdstrike.Report"
[2023-04-05 19:21:01,717] DEBUG    config  indicators_tags                             att:source="Crowdstrike.Indicators"
[2023-04-05 19:21:01,717] DEBUG    config  actors_tags                                 att:source="Crowdstrike.Actors"
[2023-04-05 19:21:01,717] DEBUG    config  unknown_mapping                             CrowdStrike:indicator:galaxy: UNATTRIBUTED
[2023-04-05 19:21:01,717] DEBUG    config  unattributed_title                          CrowdStrike Unattributed indicators:
[2023-04-05 19:21:01,717] DEBUG    config  indicator_type_title                        Indicator Type:
[2023-04-05 19:21:01,717] DEBUG    config  malware_family_title                        Malware Family:
[2023-04-05 19:21:01,717] DEBUG    config  log_duplicates_as_sightings                 True
[2023-04-05 19:21:01,717] DEBUG    config  misp_url                                    https://3samisp
[2023-04-05 19:21:01,717] DEBUG    config  misp_auth_key                               value redacted, check config file
[2023-04-05 19:21:01,717] DEBUG    config  crowdstrike_org_uuid                        ca4f4b5d-db04-4a5e-a6de-e60636dc01be
[2023-04-05 19:21:01,717] DEBUG    config  misp_enable_ssl                             False
[2023-04-05 19:21:01,717] WARNING  config  misp_enable_ssl                             SSL is disabled for MISP API requests
[2023-04-05 19:21:01,717] DEBUG    config  ind_attribute_batch_size                    2500
[2023-04-05 19:21:01,717] DEBUG    config  event_save_memory_refresh_interval          180
[2023-04-05 19:21:01,717] DEBUG    config  max_threads                                 16
[2023-04-05 19:21:01,717] DEBUG    config  miss_track_file                             no_galaxy_mapping.log
[2023-04-05 19:21:01,717] DEBUG    config  galaxies_map_file                           galaxy.ini
[2023-04-05 19:21:01,718] DEBUG    config  tag_unknown_galaxy_maps                     True
[2023-04-05 19:21:01,718] DEBUG    config  taxonomic_kill-chain                        True
[2023-04-05 19:21:01,718] DEBUG    config  taxonomic_information-security-data-source  True
[2023-04-05 19:21:01,718] DEBUG    config  taxonomic_type                              True
[2023-04-05 19:21:01,718] DEBUG    config  taxonomic_iep                               False
[2023-04-05 19:21:01,718] DEBUG    config  taxonomic_iep2                              True
[2023-04-05 19:21:01,718] DEBUG    config  taxonomic_iep2_version                      False
[2023-04-05 19:21:01,718] DEBUG    config  taxonomic_tlp                               True
[2023-04-05 19:21:01,718] DEBUG    config  taxonomic_workflow                          True
[2023-04-05 19:21:02,461] INFO     config  No configuration errors found (1 warning)
[2023-04-05 19:21:02,461] INFO     config  
[2023-04-05 19:21:02,461] INFO     config  ____ _  _ ____ ____ _  _ ____    ___  ____ ____ ____ ____ ___
[2023-04-05 19:21:02,461] INFO     config  |    |__| |___ |    |_/  [__     |__] |__| [__  [__  |___ |  \
[2023-04-05 19:21:02,461] INFO     config  |___ |  | |___ |___ | \_ ___]    |    |  | ___] ___] |___ |__/
[2023-04-05 19:21:02,461] INFO     config  
[2023-04-05 19:21:05,241] INFO     processor/main       BEGIN INDICATORS IMPORT
[2023-04-05 19:21:05,242] INFO     processor/main       Retrieving lookup data for import of CrowdStrike indicators into MISP.
[2023-04-05 19:21:05,619] INFO     processor/main       Adding 24 CrowdStrike indicator type events to MISP.
[2023-04-05 19:21:05,639] INFO     processor/main       Retrieved 0 CrowdStrike indicator malware family events from MISP.
[2023-04-05 19:21:05,728] INFO     processor/thread_8   Retrieved 309 bitcoin_address indicators from MISP.
[2023-04-05 19:21:05,736] INFO     processor/thread_12  Retrieved 495 registry indicators from MISP.
[2023-04-05 19:21:05,748] INFO     processor/thread_9   Retrieved 15 coin_address indicators from MISP.
[2023-04-05 19:21:06,002] INFO     processor/thread_3   Retrieved 1,770 hash_imphash indicators from MISP.
[2023-04-05 19:21:06,006] INFO     processor/thread_13  Retrieved 50 device_name indicators from MISP.
[2023-04-05 19:21:07,337] INFO     processor/thread_7   Retrieved 3,922 mutex_name indicators from MISP.
[2023-04-05 19:21:07,917] INFO     processor/thread_12  Retrieved 29 service_name indicators from MISP.
[2023-04-05 19:21:08,129] INFO     processor/thread_3   Retrieved 443 port indicators from MISP.
[2023-04-05 19:21:29,770] INFO     processor/thread_15  Retrieved 250 campaign_id indicators from MISP.
[2023-04-05 19:21:33,907] INFO     processor/thread_9   Retrieved 90,336 user_agent indicators from MISP.
[2023-04-05 19:21:40,469] INFO     processor/thread_4   Retrieved 201,108 file_name indicators from MISP.
[2023-04-05 19:21:45,887] INFO     processor/thread_5   Retrieved 201,108 file_path indicators from MISP.
[2023-04-05 19:21:59,117] INFO     processor/thread_2   Retrieved 412,705 hash_sha1 indicators from MISP.
[2023-04-05 19:22:08,775] INFO     processor/thread_10  Retrieved 65,980 email_address indicators from MISP.
[2023-04-05 19:23:52,617] INFO     processor/thread_14  Retrieved 1,611,349 domain indicators from MISP.
[2023-04-05 19:24:34,694] INFO     processor/thread_11  Retrieved 797,295 email_subject indicators from MISP.
[2023-04-05 19:28:23,455] INFO     processor/thread_1   Retrieved 2,807,413 hash_sha256 indicators from MISP.
[2023-04-05 19:30:26,405] INFO     processor/thread_0   Retrieved 3,351,444 hash_md5 indicators from MISP.
[2023-04-05 19:35:09,500] INFO     processor/thread_8   Retrieved 4,769,797 ip_address indicators from MISP.
[2023-04-05 20:02:58,184] INFO     processor/thread_6   Retrieved 16,365,828 url indicators from MISP.
[2023-04-05 20:03:23,340] INFO     processor/main       Found 0 pre-existing indicators within CrowdStrike reports.
[2023-04-05 20:03:51,666] INFO     processor/main       Starting import of CrowdStrike indicators into MISP.
[2023-04-05 20:05:09,132] INFO     processor/main       Retrieved 5,000 of 2,130,594 remaining indicators.
[2023-04-05 20:05:09,132] DEBUG    processor/main       Configuration states we should process batches of 2,500 indicators.
[2023-04-05 20:05:09,133] INFO     processor/main       Processing batch of 2,500 indicators.

<<>>

[2023-04-06 04:10:21,093] DEBUG    processor/thread_11  Tagged threat TARGETED
[2023-04-06 04:10:21,101] DEBUG    processor/thread_11  Added 99bfe2258817f3a9098cf69e90b2ade1f86bb0c6711ee40dcc2a5b9abc2f24ce indicators to event Malware Family: EggShellMilt
[2023-04-06 04:10:21,101] DEBUG    processor/thread_11  Creating attribute for indicator 99bfe2258817f3a9098cf69e90b2ade1f86bb0c6711ee40dcc2a5b9abc2f24ce
[2023-04-06 04:10:21,292] DEBUG    processor/thread_12  Added 698614eb8f717cb618055f8689360452ff9a3e21210b7327a3449b00ef527804 indicators to event Indicator Type: SHA256 hashes
[2023-04-06 04:10:21,292] DEBUG    processor/thread_12  Tagged malicious-confidence HIGH
[2023-04-06 04:10:21,293] DEBUG    processor/thread_12  Tagged malware CRAT
[2023-04-06 04:10:21,293] DEBUG    processor/thread_12  Tagged mitre-attck COLLECTION
[2023-04-06 04:10:21,293] DEBUG    processor/thread_12  Tagged mitre-attck COLLECTION
[2023-04-06 04:10:21,293] DEBUG    processor/thread_12  Tagged mitre-attck COLLECTION
[2023-04-06 04:10:21,293] DEBUG    processor/thread_12  Tagged mitre-attck COMMANDANDCONTROL
[2023-04-06 04:10:21,293] DEBUG    processor/thread_12  Tagged mitre-attck COMMANDANDCONTROL
[2023-04-06 04:10:21,293] DEBUG    processor/thread_12  Tagged mitre-attck DEFENSEEVASION
[2023-04-06 04:10:21,293] DEBUG    processor/thread_12  Tagged mitre-attck DEFENSEEVASION
[2023-04-06 04:10:21,293] DEBUG    processor/thread_12  Tagged mitre-attck DEFENSEEVASION
[2023-04-06 04:10:21,293] DEBUG    processor/thread_12  Tagged mitre-attck DISCOVERY
[2023-04-06 04:10:21,293] DEBUG    processor/thread_12  Tagged mitre-attck DISCOVERY
[2023-04-06 04:10:21,293] DEBUG    processor/thread_12  Tagged mitre-attck EXECUTION
[2023-04-06 04:10:21,293] DEBUG    processor/thread_12  Tagged mitre-attck EXECUTION
[2023-04-06 04:10:21,293] DEBUG    processor/thread_12  Tagged mitre-attck INITIALACCESS
[2023-04-06 04:10:21,293] DEBUG    processor/thread_12  Tagged threat TARGETED
[2023-04-06 04:10:21,296] DEBUG    processor/thread_12  Added 698614eb8f717cb618055f8689360452ff9a3e21210b7327a3449b00ef527804 indicators to event Malware Family: CRAT
[2023-04-06 04:10:21,296] DEBUG    processor/thread_12  Creating attribute for indicator 698614eb8f717cb618055f8689360452ff9a3e21210b7327a3449b00ef527804
Traceback (most recent call last):
  File "misp_import.py", line 377, in <module>
    main()
  File "misp_import.py", line 356, in main
    importer.import_from_crowdstrike(int(settings["CrowdStrike"]["init_reports_days_before"]),
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/importer.py", line 314, in import_from_crowdstrike
    self.indicators_importer.process_indicators(indicators_minutes_before)
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/indicators.py", line 195, in process_indicators
    self.push_indicators(indicators_page)
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/indicators.py", line 383, in push_indicators
    total, f_successes, f_failures, m_successes, m_failures = self.process_indicator_batch(batch)
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/indicators.py", line 325, in process_indicator_batch
    if fut.result().get("feed"):
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/indicators.py", line 225, in indicator_thread
    feed_return, fam_return = self.add_indicator_event(ind, batch_lock) # All sharing the same lock
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/indicators.py", line 528, in add_indicator_event
    itype = IndicatorType[indicator.get('type', None).upper()].value
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/enum.py", line 387, in __getitem__
    return cls._member_map_[name]
KeyError: 'PERSONA_NAME'
(.ohm) [rx118r@md2nj02di:~/src/crowdstrike/MISP-tools-main]$
jshcodes commented 1 year ago

This looks to be related to a failed lookup in the IndicatorType enum. I've not seen the PERSONA_NAME indicator type before. Investigating where this originated from.

netscylla commented 1 year ago

Lazy fix:

Added this line to cs_misp_import/indicator_type.py

PERSONA_NAME = "Persona / alias"

jshcodes commented 1 year ago

Lazy fix:

Added this line to cs_misp_import/indicator_type.py

PERSONA_NAME = "Persona / alias"

Adding this new type to the enum is the right solution. Adding this to the next revision.