search

OpenCTI-Platform / connectors

OpenCTI Connectors
https://www.opencti.io
Apache License 2.0
360 stars 391 forks source link

Malware Bazar Connector Error #2114

Open 1337Deep opened 3 months ago

1337Deep commented 3 months ago

Description

Got this Error

ERROR {'name': 'MISSING_REFERENCE_ERROR', 'message': 'Element(s) not found'} | timestamp=2024-05-13T06:03:03.679354Z name=MalwareBazaar Recent Additions exc_info=Traceback (most recent call last): File "/opt/opencti-connector-malwarebazaar-recent-additions/malwarebazaar-recent-additions.py", line 128, in run response = self.upload_artifact_opencti( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/opencti-connector-malwarebazaar-recent-additions/malwarebazaar-recent-additions.py", line 260, in upload_artifact_opencti return self.helper.api.stix_cyber_observable.upload_artifact(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/pycti/entities/opencti_stix_cyber_observable.py", line 1802, in upload_artifact result = self.opencti.query( ^^^^^^^^^^^^^^^^^^^

Environment

OS (where OpenCTI server runs): {Ubuntu 23.10} OpenCTI version: { 6.1.0 } OpenCTI client: { python }

Reproducible Steps

Steps to create the smallest reproducible scenario: Steps to create the smallest reproducible scenario: open Malware Bazar Connector Logs

Expected Output

Actual Output

raceback (most recent call last):
  File "/opt/opencti-connector-malwarebazaar-recent-additions/malwarebazaar-recent-additions.py", line 90, in run
    recent_additions_list = self.get_recent_additions()
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/opencti-connector-malwarebazaar-recent-additions/malwarebazaar-recent-additions.py", line 191, in get_recent_additions
    resp = requests.post(self.api_url, data=data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='mb-api.abuse.ch', port=443): Max retries exceeded with url: /api/v1/ (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7c6b3e3e24d0>: Failed to resolve 'mb-api.abuse.ch' ([Errno -3] Try again)"))
ERROR {'name': 'MISSING_REFERENCE_ERROR', 'message': 'Element(s) not found'} | timestamp=2024-05-13T06:13:10.949042Z name=MalwareBazaar Recent Additions exc_info=Traceback (most recent call last):
  File "/opt/opencti-connector-malwarebazaar-recent-additions/malwarebazaar-recent-additions.py", line 128, in run
    response = self.upload_artifact_opencti(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/opencti-connector-malwarebazaar-recent-additions/malwarebazaar-recent-additions.py", line 260, in upload_artifact_opencti
    return self.helper.api.stix_cyber_observable.upload_artifact(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pycti/entities/opencti_stix_cyber_observable.py", line 1802, in upload_artifact
    result = self.opencti.query(
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py", line 353, in query
    raise ValueError(
ValueError: {'name': 'MISSING_REFERENCE_ERROR', 'message': 'Element(s) not found'}

Additional information

Screenshots (optional)

Jipegien commented 3 months ago

@ckane I see that you participate to the code "recently". Do you have any idea about how to fix this? For me it might be that some entities are not created before a relationship that link them but I might be wrong.

ckane commented 3 months ago

Hi there - apologies for the delay in getting back on this. I wanted to update to 6.1.2 from 6.0.10 before looking into it. I have not observed this behavior on my instance before or since updating, so I am not able to make an educated guess based just upon the errors above. It appears there are two errors in the "Actual Output" - first is a failure to connect to abuse.ch, and second is the MISSING_REFERENCE_ERROR. The timestamp wasn't included in the connection failure error, so I don't have any way to determine how close these two occurred in time to each other.

It is entirely possible that there could have been a platform update/outage/bug occurring upstream at Malware Bazaar, and the error is a result of corrupted/incomplete data attempted to be consumed from Malware Bazaar during the course of this event.

Would have to see more examples of this happening to get an idea for what is going on. That said, from my experience the Malware Bazaar Recent Additions connector seems to fetch all of the recent additions of the past 60 minutes, regardless of the last connector state (https://github.com/OpenCTI-Platform/connectors/blob/master/external-import/malwarebazaar-recent-additions/src/malwarebazaar-recent-additions.py#L187), and then will try to determine if the entity was already consumed into OpenCTI by performing a query (https://github.com/OpenCTI-Platform/connectors/blob/master/external-import/malwarebazaar-recent-additions/src/malwarebazaar-recent-additions.py#L117) before ingest. In this particular case, the MISSING_REFERENCE_ERROR in the upload_artifact call suggests that the upload failed and the entity was not constructed (correct me if this assumption is a wrong interpretation of the error message).

So, hypothetically if this occurred once and the connector is configured to attempt a pull multiple times per hour, and then successive attempts to pull recent additions don't fail, then the entities in question can reasonably be presumed to have been consumed successfully.

If you set the log level to info instead of error for the connector, then if this error keeps occurring over and over, the connector will also report above it which item from Malware Bazaar was attempting to be ingested at that time (https://github.com/OpenCTI-Platform/connectors/blob/master/external-import/malwarebazaar-recent-additions/src/malwarebazaar-recent-additions.py#L92), and in that case the raw data from MBRA can be pulled down and we can look at what is missing from the data.