search

OpenCTI-Platform / client-python

OpenCTI Python Client
https://www.opencti.io
Apache License 2.0
115 stars 129 forks source link

Event stream synchronization does not process additional_names or external_references for Artifacts #265

Closed kmcmahon1959 closed 2 years ago

kmcmahon1959 commented 2 years ago

When ingesting SSE data from a local file (using a modified version of local_synchronizer.py) the additional_names data and external_references are not processed on the target system.

Environment

  1. OS (where OpenCTI server runs): Ubuntu 20.4
  2. OpenCTI version: 5.3.0
  3. Other environment details: dockerized version

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Connect to raw stream&from= saving events to a file
  2. use modified local_syncrhonizer on target system to read file and import bundles.
  3. Screen shots below show the difference between the two systems.

Expected Output

Screen shot 1 shows a particular artifact observable on the source system. Note that this image shows the additional_names and external_references data.

Actual Output

Screen 2 show the same artifact observable on the target system. Note that this screen does not show the additional_names nor external references for the same artifact.

Additional information

The associated SSE events for this issue are listed below; note that the additional_names and external_references data is included in the SSE capture below:

id: 1656470639460-0 event: create data: {"version":"4","type":"create","message":"creates a Artifact 666c6f2f3993946d4236f3ce33d105e144631e704077b676dd592c19024f57ae","origin":{"ip":"::ffff:172.18.0.10","user_id":"88ec0c6a-13ce-5e39-b486-354fe4a7084f"},"data":{"id":"artifact--d960b001-63c1-5944-8c1c-7ca97ff76fec","spec_version":"2.1","type":"artifact","extensions":{"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba":{"extension_type":"property-extension","id":"f6330c55-3c8b-409d-82fe-738e06c492c1","type":"Artifact","created_at":"2022-06-29T02:43:59.409Z","updated_at":"2022-06-29T02:43:59.409Z","is_inferred":false},"extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82":{"extension_type":"property-extension","description":"Uploaded to MalwareBazaar by Twitter user: SecuriteInfoCom.","additional_names":["SecuriteInfo.com.W32.AIDetectNet.01.31069.12377"]}},"mime_type":"application/x-dosexec","hashes":{"MD5":"fea5486bd37725bb4e8ca67caac79d39","SHA-1":"9a34b0d02b41e0ab92a7ab7f038d3edaf79022a0","SHA-256":"666c6f2f3993946d4236f3ce33d105e144631e704077b676dd592c19024f57ae"}}}

id: 1656470639875-0 event: create data: {"version":"4","type":"create","message":"creates a External-Reference MalwareBazaar Recent Additions","origin":{"ip":"::ffff:172.18.0.10","user_id":"88ec0c6a-13ce-5e39-b486-354fe4a7084f"},"data":{"id":"external-reference--02628a6f-1389-51c2-a118-24dd3a3457fc","spec_version":"2.1","type":"external-reference","extensions":{"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba":{"extension_type":"new-sdo","id":"e2b4adc0-c2e9-4b78-851e-ded0f8a91f9d","type":"External-Reference","created_at":"2022-06-29T02:43:59.583Z","updated_at":"2022-06-29T02:43:59.583Z","is_inferred":false}},"source_name":"MalwareBazaar Recent Additions","description":"MalwareBazaar Recent Additions","url":"https://bazaar.abuse.ch/sample/666c6f2f3993946d4236f3ce33d105e144631e704077b676dd592c19024f57ae/"}}

id: 1656470640159-0 event: update data: {"version":"4","type":"update","message":"adds MalwareBazaar Recent Additions in external_references","origin":{"ip":"::ffff:172.18.0.10","user_id":"88ec0c6a-13ce-5e39-b486-354fe4a7084f"},"data":{"id":"artifact--d960b001-63c1-5944-8c1c-7ca97ff76fec","spec_version":"2.1","type":"artifact","extensions":{"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba":{"extension_type":"property-extension","id":"f6330c55-3c8b-409d-82fe-738e06c492c1","type":"Artifact","created_at":"2022-06-29T02:43:59.409Z","updated_at":"2022-06-29T02:43:59.409Z","is_inferred":false},"extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82":{"extension_type":"property-extension","description":"Uploaded to MalwareBazaar by Twitter user: SecuriteInfoCom.","external_references":[{"source_name":"MalwareBazaar Recent Additions","description":"MalwareBazaar Recent Additions","url":"https://bazaar.abuse.ch/sample/666c6f2f3993946d4236f3ce33d105e144631e704077b676dd592c19024f57ae/"}],"additional_names":["SecuriteInfo.com.W32.AIDetectNet.01.31069.12377"]}},"mime_type":"application/x-dosexec","hashes":{"MD5":"fea5486bd37725bb4e8ca67caac79d39","SHA-1":"9a34b0d02b41e0ab92a7ab7f038d3edaf79022a0","SHA-256":"666c6f2f3993946d4236f3ce33d105e144631e704077b676dd592c19024f57ae"}},"context":{"patch":[{"op":"add","path":"/extensions/extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82/external_references","value":[{"source_name":"MalwareBazaar Recent Additions","description":"MalwareBazaar Recent Additions","url":"https://bazaar.abuse.ch/sample/666c6f2f3993946d4236f3ce33d105e144631e704077b676dd592c19024f57ae/"}]}],"reverse_patch":[{"op":"remove","path":"/extensions/extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82/external_references"}]}}

id: 1656470640547-0 event: update data: {"version":"4","type":"update","message":"adds malware-bazar in labels","origin":{"ip":"::ffff:172.18.0.10","user_id":"88ec0c6a-13ce-5e39-b486-354fe4a7084f"},"data":{"id":"artifact--d960b001-63c1-5944-8c1c-7ca97ff76fec","spec_version":"2.1","type":"artifact","extensions":{"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba":{"extension_type":"property-extension","id":"f6330c55-3c8b-409d-82fe-738e06c492c1","type":"Artifact","created_at":"2022-06-29T02:43:59.409Z","updated_at":"2022-06-29T02:44:00.118Z","is_inferred":false},"extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82":{"extension_type":"property-extension","labels":["malware-bazar"],"description":"Uploaded to MalwareBazaar by Twitter user: SecuriteInfoCom.","external_references":[{"source_name":"MalwareBazaar Recent Additions","description":"MalwareBazaar Recent Additions","url":"https://bazaar.abuse.ch/sample/666c6f2f3993946d4236f3ce33d105e144631e704077b676dd592c19024f57ae/"}],"additional_names":["SecuriteInfo.com.W32.AIDetectNet.01.31069.12377"]}},"mime_type":"application/x-dosexec","hashes":{"MD5":"fea5486bd37725bb4e8ca67caac79d39","SHA-1":"9a34b0d02b41e0ab92a7ab7f038d3edaf79022a0","SHA-256":"666c6f2f3993946d4236f3ce33d105e144631e704077b676dd592c19024f57ae"}},"context":{"patch":[{"op":"add","path":"/extensions/extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82/labels","value":["malware-bazar"]}],"reverse_patch":[{"op":"remove","path":"/extensions/extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82/labels"}]}}

id: 1656470640817-0 event: update data: {"version":"4","type":"update","message":"adds agenttesla in labels","origin":{"ip":"::ffff:172.18.0.10","user_id":"88ec0c6a-13ce-5e39-b486-354fe4a7084f"},"data":{"id":"artifact--d960b001-63c1-5944-8c1c-7ca97ff76fec","spec_version":"2.1","type":"artifact","extensions":{"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba":{"extension_type":"property-extension","id":"f6330c55-3c8b-409d-82fe-738e06c492c1","type":"Artifact","created_at":"2022-06-29T02:43:59.409Z","updated_at":"2022-06-29T02:44:00.519Z","is_inferred":false},"extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82":{"extension_type":"property-extension","labels":["malware-bazar","agenttesla"],"description":"Uploaded to MalwareBazaar by Twitter user: SecuriteInfoCom.","external_references":[{"source_name":"MalwareBazaar Recent Additions","description":"MalwareBazaar Recent Additions","url":"https://bazaar.abuse.ch/sample/666c6f2f3993946d4236f3ce33d105e144631e704077b676dd592c19024f57ae/"}],"additional_names":["SecuriteInfo.com.W32.AIDetectNet.01.31069.12377"]}},"mime_type":"application/x-dosexec","hashes":{"MD5":"fea5486bd37725bb4e8ca67caac79d39","SHA-1":"9a34b0d02b41e0ab92a7ab7f038d3edaf79022a0","SHA-256":"666c6f2f3993946d4236f3ce33d105e144631e704077b676dd592c19024f57ae"}},"context":{"patch":[{"op":"add","path":"/extensions/extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82/labels/1","value":"agenttesla"}],"reverse_patch":[{"op":"remove","path":"/extensions/extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82/labels/1"}]}}

id: 1656470641049-0 event: update data: {"version":"4","type":"update","message":"adds exe in labels","origin":{"ip":"::ffff:172.18.0.10","user_id":"88ec0c6a-13ce-5e39-b486-354fe4a7084f"},"data":{"id":"artifact--d960b001-63c1-5944-8c1c-7ca97ff76fec","spec_version":"2.1","type":"artifact","extensions":{"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba":{"extension_type":"property-extension","id":"f6330c55-3c8b-409d-82fe-738e06c492c1","type":"Artifact","created_at":"2022-06-29T02:43:59.409Z","updated_at":"2022-06-29T02:44:00.792Z","is_inferred":false},"extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82":{"extension_type":"property-extension","labels":["malware-bazar","agenttesla","exe"],"description":"Uploaded to MalwareBazaar by Twitter user: SecuriteInfoCom.","external_references":[{"source_name":"MalwareBazaar Recent Additions","description":"MalwareBazaar Recent Additions","url":"https://bazaar.abuse.ch/sample/666c6f2f3993946d4236f3ce33d105e144631e704077b676dd592c19024f57ae/"}],"additional_names":["SecuriteInfo.com.W32.AIDetectNet.01.31069.12377"]}},"mime_type":"application/x-dosexec","hashes":{"MD5":"fea5486bd37725bb4e8ca67caac79d39","SHA-1":"9a34b0d02b41e0ab92a7ab7f038d3edaf79022a0","SHA-256":"666c6f2f3993946d4236f3ce33d105e144631e704077b676dd592c19024f57ae"}},"context":{"patch":[{"op":"add","path":"/extensions/extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82/labels/2","value":"exe"}],"reverse_patch":[{"op":"remove","path":"/extensions/extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82/labels/2"}]}}

artifact-observable-666c6f-source-system artifact-observable-666c6f-target-system

kmcmahon1959 commented 2 years ago

Note also, that the container running the import on the target system is running pycti 5.3.7.

kmcmahon1959 commented 2 years ago

Testing this today and now every Artifact ingest on the target system fails with the following error: INFO:root:Processing event 1661979904983-0 INFO:root:Creating Stix-Cyber-Observable {Artifact} with indicator at False. ERROR:root:ERROR: could not process message id: 1661979904983-0 Traceback (most recent call last): File "/opt/opencti-highside-sync/read-stream-from-file.py", line 68, in _process_message self.opencti_target_client.stix2.import_bundle(bundle, True) File "/usr/local/lib/python3.10/site-packages/pycti/utils/opencti_stix2.py", line 1835, in import_bundle self.import_observable(item, update, types) File "/usr/local/lib/python3.10/site-packages/pycti/utils/opencti_stix2.py", line 777, in import_observable data=base64.b64decode(file["data"]), KeyError: 'data'

...because there is no "data" element in the bundle.