search

OpenCTI-Platform / opencti

Open Cyber Threat Intelligence Platform
https://opencti.io
Other
6.11k stars 906 forks source link

URLHaus Connector Import Errors #2181

Open apmillett opened 2 years ago

apmillett commented 2 years ago

Description

Having added the URLHaus connector, we're seeing lots of errors importing data on the connector. For example -

Jun 21, 2022, 9:08:53 PM {'name': 'UnsupportedError', 'message': 'Input resolve refs expect single value'} {"type": "bundle", "id": "bundle--f7376d90-f22a-4a93-8db8-3fe734825d23", "spec_version": "2.1", "x_opencti_seq": 1, "objects": [{"type": "url", "spec_version": "2.1", "id": "url--ce742784-055e-58f0-8e26-3be27bb1b3b7", "value": "https://drive.google.com/uc?export=download&id=1By94bArhLEiBjU_aAhJFJsxd6HGLhPYR&confirm=t", "object_marking_refs": ["marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9"], "created_by_ref": "identity--0303206b-ec74-5e9e-81df-e6532e9c1e91", "description": "Threat: malware_download - Reporter: Cryptolaemus1 - Status: offline", "external_references": [{"source_name": "Abuse.ch URLhaus", "description": "URLhaus repository URL", "url": "https://urlhaus.abuse.ch/url/2246566/"}], "labels": ["aa", "E98346", "html", "Qakbot", "qbot", "TR", "zip"], "x_opencti_create_indicator": true, "x_opencti_score": 80, "nb_deps": 1}]}

Out of the current ingest of 40641 events, 30335 events are marked as 'Unsupported' similar to above.

Environment

  1. OS (where OpenCTI server runs): Ubuntu 21.04
  2. OpenCTI version: 5.3.6
  3. OpenCTI client:
  4. Other environment details: Using the pre-bundled OVA template on VMWare.

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Add URLHaus Connector

Expected Output

Data imported successfully

Actual Output

Jun 21, 2022, 9:08:53 PM {'name': 'UnsupportedError', 'message': 'Input resolve refs expect single value'} {"type": "bundle", "id": "bundle--f7376d90-f22a-4a93-8db8-3fe734825d23", "spec_version": "2.1", "x_opencti_seq": 1, "objects": [{"type": "url", "spec_version": "2.1", "id": "url--ce742784-055e-58f0-8e26-3be27bb1b3b7", "value": "https://drive.google.com/uc?export=download&id=1By94bArhLEiBjU_aAhJFJsxd6HGLhPYR&confirm=t", "object_marking_refs": ["marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9"], "created_by_ref": "identity--0303206b-ec74-5e9e-81df-e6532e9c1e91", "description": "Threat: malware_download - Reporter: Cryptolaemus1 - Status: offline", "external_references": [{"source_name": "Abuse.ch URLhaus", "description": "URLhaus repository URL", "url": "https://urlhaus.abuse.ch/url/2246566/"}], "labels": ["aa", "E98346", "html", "Qakbot", "qbot", "TR", "zip"], "x_opencti_create_indicator": true, "x_opencti_score": 80, "nb_deps": 1}]}

Additional information

We have the following connectors installed -

Screenshot 2022-06-22 at 08 56 04

Screenshots (optional)

Screenshot 2022-06-22 at 08 22 33
SamuelHassine commented 2 years ago

Hello @apmillett,

Thanks a lot for opening this issue. The problem here is apparently something went wrong in your platform and one of the "dependencies" used in the entities ingested by URLhaus seems to be duplicated which is something that should never happen (we are working on reducing this at maximum but in some rare cases it could still happen). This is the meaning of the error `'Input resolve refs expect single value'.

Dependencies are the items that can be labels, created_by_ref, object_marking_refs, etc. My guess here is this concerns one of the label (because if it was created_by_ref, 100% of created entities would be in error, same for object_marking).

So, can you please go in the Settings => Attributes => Labels section and try to search these labels:

["aa", "E98346", "html", "Qakbot", "qbot", "TR", "zip"].

One of them is certainly duplicated. If you find it, delete just of the label and relaunch your ingestion, everything should now work.

If it's not a label, thing about search the IDs contained in created_by_refand òbject_marking_refs` in the gloal search bar surrounded by double quotes to find duplicated IDs.

Let me know your findings. You can also reach me directly on Slack so we can try to fix it together.

Kind regards, Samuel