OpenCTI-Platform / opencti

Open Cyber Threat Intelligence Platform
https://opencti.io
Other
6.07k stars 903 forks source link

Issues uploading pdfs to OpenCTI #5947

Closed Pez721 closed 5 months ago

Pez721 commented 6 months ago

Description

We are having several issues uploading documents to OpenCTI in the GUI. First, sometime when we attempt to upload reports with pdf attachments in Analyses, the wheel spins indefinitely and the Report is never created. This is related to the files, a single file will always either upload immediately or spin indefinitely. Sometimes I can take a pdf that doesnt upload, open it in word and resave as a pdf again, then it will load. Other times I can't find any way of uploading. When it fails to upload nothing is logged in opencti. I have attached an example of a file that does not upload - this one came from a website extract, but there are reports from proper vendors that fail too. Dark Web Profile NoName057(16) - SOCRadar® Cyber Intelligence Inc.pdf

Secondly, sometimes when a file is uploaded and I click "Launch an import of this file" using the import document connector, I receive an error "Invalid value for Relationship 'target_ref': not a valid STIX identifier". Example: [Feb 6, 2024, 3:47:59 PM] Invalid value for Relationship 'target_ref': not a valid STIX identifier, must match --: {'id': '078827d7-9f29-43dc-bd4b-ab5acdb3c191', 'standard_id': 'threat-actor--5d9de760-e18c-5daa-8a8c-a6e0bbbf4d15', 'entity_type': 'Threat-Actor-Group', 'parent_types': ['Basic-Object', 'Stix-Object', 'Stix-Core-Object', 'Stix-Domain-Object', 'Threat-Actor'], 'spec_version': '2.1', 'created_at': '2024-02-06T08:53:28.857Z', 'updated_at': '2024-02-06T15:47:39.499Z', 'createdBy': {'id': '5c108cdb-4c88-46da-b066-352fb6ce2518', 'standard_id': 'identity--631ca45b-2610-58cf-92c3-0697d2b3634f', 'entity_type': 'Individual', 'parent_types': ['Basic-Object', 'Stix-Object', 'Stix-Core-Object', 'Stix-Domain-Object', 'Identity'], 'spec_version': '2.1', 'identity_class': 'individual', 'name': 'xxxxx', 'description': '', 'roles': None, 'contact_information': None, 'x_opencti_aliases': None, 'created': '2024-02-06T08:50:27.302Z', 'modified': '2024-02-06T08:50:27.302Z', 'objectLabel': [], 'x_opencti_firstname': None, 'x_opencti_lastname': None, 'objectLabelIds': []}, 'objectMarking': [{'id': 'fe053437-7bb2-4ee8-aa7d-d7973651f53d', 'standard_id': 'marking-definition--f88d31f6-486f-44da-b317-01333bde0b82', 'entity_type': 'Marking-Definition', 'definition_type': 'TLP', 'definition': 'TLP:AMBER', 'created': '2023-09-29T08:51:53.552Z', 'modified': '2024-01-26T14:27:33.437Z', 'x_opencti_order': 3, 'x_opencti_color': '#d84315', 'createdById': None}], 'objectLabel': [{'id': '3bae34cc-a9a1-4e19-8085-44f92f5b5a58', 'value': 'turkey', 'color': '#ae5698', 'createdById': None}, {'id': 'dcfce0ad-5580-4de5-9669-e99b3dd96f9e', 'value': 'telecommunications', 'color': '#87bd48', 'createdById': None}, {'id': '333852ad-ded1-431d-989f-5e84e1c36388', 'value': 'crowdstrike', 'color': '#9f0242', 'createdById': None}], 'externalReferences': [{'id': '6590f0a2-a9fc-435b-bb86-fcdb77c1cafb', 'standard_id': 'external-reference--4b56e9ad-6684-58bf-8f60-6fc29cda6deb', 'entity_type': 'External-Reference', 'source_name': 'Recent PwnKit (CVE-2021-4034) Exploitation Represents an Enduring Threat; Targeting of Telecommunications Entities Continues', 'description': 'PwnKit is a LPE vulnerability (CVE-2021-4034) impacting the Polkit component of many Linux/Unix distributions.\nSpecifically, this out-of-bounds write vulnerability allows an adversary to bypass restrictions and insert an\nenvironment variable, which executes a specifically crafted payload with root permissions via pkexec.', 'url': 'https://falcon.us-2.crowdstrike.com/api2/intel/reports/entities/report-files/v1?ids=189243', 'hash': None, 'external_id': 'csa-221042-recent-pwnkit-cve-2021-4034-exploitation-represen...', 'created': '2024-02-06T15:47:39.084Z', 'modified': '2024-02-06T15:47:39.084Z', 'createdById': None}], 'revoked': False, 'confidence': 84, 'created': '2024-02-06T08:53:28.857Z', 'modified': '2024-02-06T15:47:39.499Z', 'name': 'Sea Turtle', 'description': 'Sea Turtle, known for its alignment with Turkish interests, operates within multiple aliases, including Teal Kurma, Marbled Dust, SILICON, Cosmic Wolf and UNC1326. This is an espionage-driven Advanced Persistent Threat (APT) actor, and it is derived towards the public and private sectors, with the highest level of interest expressed in relation to telecommunication entities, ISPs, IT service providers, and media organizations.', 'aliases': ['Teal Kurma', 'Marbled Dust', 'SILICON', 'Cosmic Wolf', 'UNC1326'], 'threat_actor_types': ['nation-state'], 'first_seen': '2017-01-06T00:00:00.000Z', 'last_seen': '2024-01-06T00:00:00.000Z', 'roles': ['malware-author', 'director'], 'goals': [], 'sophistication': 'expert', 'resource_level': 'government', 'primary_motivation': 'organizational-gain', 'secondary_motivations': ['ideology'], 'personal_motivations': ['dominance'], 'createdById': '5c108cdb-4c88-46da-b066-352fb6ce2518', 'objectMarkingIds': ['fe053437-7bb2-4ee8-aa7d-d7973651f53d'], 'objectLabelIds': ['3bae34cc-a9a1-4e19-8085-44f92f5b5a58', 'dcfce0ad-5580-4de5-9669-e99b3dd96f9e', '333852ad-ded1-431d-989f-5e84e1c36388'], 'externalReferencesIds': ['6590f0a2-a9fc-435b-bb86-fcdb77c1cafb']}

Finally, when the upload of a file and its import were successful, I go to the analyst workbench and click "Validate Workbench". Sometimes this hangs with no error logged and the workbench never validates.

Any help you can give on these issues would be much appreciated.

Environment

  1. OS (where OpenCTI server runs): linux containerd
  2. OpenCTI version: OpenCTI 5.12.23
  3. OpenCTI client: frontend
  4. Other environment details: running in kubernetes.
nino-filigran commented 6 months ago

@Pez721 let me try to undertand the different issues here.

Could you pelase ensure that you're running on the most up to date version and tell we what roles (and rights associated to your role) you have?

Pez721 commented 6 months ago

Thanks for your quick response. I'll try and provide as much detail as possible.

I'm using the admin account, so there should not be any permissions problems. I have upgraded opencti to the latest 5.12.30 release and the problem still occurs. Minio is running RELEASE.2023-06-23T20-26-00Z RabbitMQ is running version: 3.12.2 Redis is hosted by google cloud (memorystore) and running version 7.0

Lets focus on the first issue, the upload of files. It doesn't matter which approach I take, some files are never uploaded into the system. For example, if I click "Data import and analyst workbenches" then "Select your file" and select the file I attached to you, the wheel spins indefinitely (see image attached). If I click Analyses, the add icon in bottom right, select your file, then "Create", the create button greys out but the window doesn't close and the report is never created.

openctiFileImport

It would be useful to know what is going on behind the scenes when a file is uploaded. I assume it is stored in Minio? I went and looked in the minio console and I saw there was no difference in the number of objects present before attempting to upload a failing file and afterwards. Does Opencti upload the file to Minio directly or does it happen via a worker pod or rabbitmq task?

Thanks for your help

nino-filigran commented 6 months ago

@Pez721 I'm not technical myself and I cannot answer your questions. @Kedae do you know the anwsers to these questions?

Pez721 commented 6 months ago

Any chance you could look at this?

nino-filigran commented 5 months ago

@Pez721 I see that there was not any activity on this ticket since then. Do you still encounter some issues?

nino-filigran commented 5 months ago

Closing this ticket due to inactivity. If you have any other question or would like to re-open the ticket, feel free to do it.

Regards