Issues with Actor Import (Script Failure on KeyError)

[roofdiver]

Hi,

Every few months there seems to be an issue with Actor Importing that causes the script to crash. The first time it was Sphynx which I see was updated. Today the issue is warnings / error DRAGNET/OCTANE Panda and a script crash with a Key Error for SAIGA.

Is there a solid solution to make sure that this doesn't keep breaking or some way to make sure the actor list is dynamically updated?

If I am missing something, please let me know. This is a really great tool and it works well for us most of the time.

On the latest update (0.7.3)

[2024-05-08 15:11:33,733] WARNING  processor/thread_7   Adversary OCTANE PANDA missing field first_activity_date.
[2024-05-08 15:11:42,466] WARNING  processor/thread_8   Adversary DRAGNET PANDA missing field first_activity_date.
[2024-05-08 15:11:42,467] WARNING  processor/thread_8   Adversary DRAGNET PANDA missing field last_activity_date.
[2024-05-08 15:11:44,910] WARNING  processor/thread_8   Could not add or tag event ADV-225759 DRAGNET PANDA (People's Republic of China). Will retry in 0.3 seconds.
day is out of range for month: 0
Traceback (most recent call last):
  File "<redacted>/misp_import.py", line 408, in <module>
    main()
  File "<redacted>/misp_import.py", line 387, in main
    importer.import_from_crowdstrike(int(settings["CrowdStrike"]["init_reports_days_before"]),
  File "<redacted>cs_misp_import/importer.py", line 342, in import_from_crowdstrike
    self.actors_importer.process_actors(actors_days_before, self.event_ids)
  File "<redacted>/cs_misp_import/actors.py", line 223, in process_actors
    if fut.result():
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "<redacted>", line 79, in batch_import_actors
    act_detail = Adversary[actor_name.split(" ")[1].upper()].value
  File "/usr/lib/python3.10/enum.py", line 440, in __getitem__
    return cls._member_map_[name]
KeyError: 'SAIGA'

[avinashKumarYadav]

Same here i am also facing this same issue, tried in both production and test instances.

[2024-05-16 18:54:38,510] WARNING processor/thread_5 Could not add or tag event ADV-224558 SLY SPIDER (eCrime). Will retry in 0.3 seconds. Error code 500: An Internal Error Has Occurred. [2024-05-16 18:54:38,790] WARNING processor/thread_3 Could not add or tag event ADV-209074 TUNNEL SPIDER (eCrime). Will retry in 0.3 seconds. Error code 500: An Internal Error Has Occurred. Traceback (most recent call last): File "/home/ayadav3/MISP-tools/misp_import.py", line 408, in main() File "/home/ayadav3/MISP-tools/misp_import.py", line 387, in main importer.import_from_crowdstrike(int(settings["CrowdStrike"]["init_reports_days_before"]), File "/home/ayadav3/MISP-tools/cs_misp_import/importer.py", line 342, in import_from_crowdstrike self.actors_importer.process_actors(actors_days_before, self.event_ids) File "/home/ayadav3/MISP-tools/cs_misp_import/actors.py", line 223, in process_actors if fut.result(): File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result return self.get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/home/ayadav3/MISP-tools/cs_misp_import/actors.py", line 79, in batch_import_actors act_detail = Adversary[actor_name.split(" ")[1].upper()].value File "/usr/lib/python3.10/enum.py", line 440, in getitem return cls._membermap[name] KeyError: 'SAIGA'

[roofdiver]

I have a temporary solution that can solve the error I can share tomorrow, but the deeper issue is extensive use of static enums for data that is dynamic.

in this case, the error is caused because the script is searching for country attribution for SAIGA, but the adversaries enum (not at my computer so will comment tomorrow with full details / file names) does not have a country for SAIGA and it crashes.

A long term solution could be to implement a mechanism (moving away from static enums) to handle new attribution as new data comes in. Rather than just waiting for a new error and then updating the enum. The same thing happened with Sphinx a while back.

[roofdiver]

Okay so temporary solution is to add SAIGA attribution to this enum.

according to https://www.crowdstrike.com/adversaries/comrade-saiga/ it would be kazakhstan.

[roofdiver]

tagging dev @jshcodes for visibility

[jshcodes]

The adversary branch enumerator has been updated and released in version 0.7.4.

[roofdiver]

@jshcodes is it possible to work out a way where the script wont break when new adversary info comes through crowdstrike? from a production perspective it makes it difficult to rely on the tool when new incoming data breaks it, and then we either have to do a code change or wait for an update.