search

OpenCTI-Platform / connectors

OpenCTI Connectors
https://www.opencti.io
Apache License 2.0
388 stars 417 forks source link

ALIENVAULT connector is not working properly #1180

Open iman006 opened 1 year ago

iman006 commented 1 year ago

Description

Alienvault Connector which worked correctly and had no issues in previous versions, is not importing any feeds to the platform in the new version. no matter what changes I make in the config, the problem persists, and it doesn't make any difference when I test it with different Alienvault accounts. If anyone knows a solution to this problem, I would appreciate it if they could share it.

Environment

OS (where OpenCTI server runs): { ubuntu 20.04 } OpenCTI version: { OpenCTI 5.7.4 } OpenCTI client: { 5.7.4 } Alienvault connector version: { 5.7.4 }

Expected Output

Receive feeds from Alienvault as before and store them in the opencti platform.

Actual Output

No feed is being received and stored in the system

Docker-Compose

connector-alienvault: image: opencti/connector-alienvault:5.7.4 environment:

Additional information

{"timestamp": "2023-06-07T10:44:18.775042Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://opencti:8080 \"POST /graphql HTTP/1.1\" 200 2564"} {"timestamp": "2023-06-07T10:44:20.042708Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://opencti:8080 \"POST /graphql HTTP/1.1\" 200 606"} {"timestamp": "2023-06-07T10:44:20.061883Z", "level": "INFO", "name": "pycti.connector", "message": "Connector registered with ID: f817b5b7-0f0c-435f-b37e-b3216f2e2eab"} {"timestamp": "2023-06-07T10:44:20.062413Z", "level": "INFO", "name": "pycti.connector", "message": "Starting ping alive thread"} {"timestamp": "2023-06-07T10:44:20.065860Z", "level": "INFO", "name": "pycti.connector", "message": "Starting AlienVault connector..."} {"timestamp": "2023-06-07T10:44:20.065978Z", "level": "INFO", "name": "pycti.connector", "message": "Running AlienVault connector..."} {"timestamp": "2023-06-07T10:44:20.066347Z", "level": "INFO", "name": "pycti.connector", "message": "Loaded state: {}"} {"timestamp": "2023-06-07T10:44:20.066457Z", "level": "INFO", "name": "pycti.connector", "message": "Connector first run"} {"timestamp": "2023-06-07T10:44:20.066718Z", "level": "INFO", "name": "pycti.api", "message": "Initiate work for f817b5b7-0f0c-435f-b37e-b3216f2e2eab"} {"timestamp": "2023-06-07T10:44:20.067884Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Starting new HTTP connection (2): opencti:8080"} {"timestamp": "2023-06-07T10:44:20.729558Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://opencti:8080 \"POST /graphql HTTP/1.1\" 200 97"} {"timestamp": "2023-06-07T10:44:20.730689Z", "level": "INFO", "name": "pycti.connector", "message": "Running pulse importer (update data: True, guess malware: True, guess cve: True, relationships: True, patterns_indicates: True, filter_indicators: True)..."} {"timestamp": "2023-06-07T10:44:20.732405Z", "level": "INFO", "name": "pycti.connector", "message": "Fetching subscribed pulses..."} {"timestamp": "2023-06-07T10:44:20.734030Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Starting new HTTPS connection (1): otx.alienvault.com:443"} {"timestamp": "2023-06-07T10:44:21.281923Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://opencti:8080 \"POST /graphql HTTP/1.1\" 200 98"} {"timestamp": "2023-06-07T10:44:55.833928Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "https://otx.alienvault.com:443 \"GET /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00 HTTP/1.1\" 200 4765517"} {"timestamp": "2023-06-07T10:45:01.285376Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Resetting dropped connection: opencti"} {"timestamp": "2023-06-07T10:45:02.096742Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://opencti:8080 \"POST /graphql HTTP/1.1\" 200 98"} {"timestamp": "2023-06-07T10:45:42.099510Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Resetting dropped connection: opencti"} {"timestamp": "2023-06-07T10:45:43.218349Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://opencti:8080 \"POST /graphql HTTP/1.1\" 200 98"} {"timestamp": "2023-06-07T10:46:02.011152Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "https://otx.alienvault.com:443 \"GET /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2 HTTP/1.1\" 504 132"} {"timestamp": "2023-06-07T10:46:02.011727Z", "level": "DEBUG", "name": "urllib3.util.retry", "message": "Incremented Retry for (url='/api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2'): Retry(total=4, connect=None, read=None, redirect=None, status=None)"} {"timestamp": "2023-06-07T10:46:02.012091Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Retry: /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2"} {"timestamp": "2023-06-07T10:46:03.266763Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "https://otx.alienvault.com:443 \"GET /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2 HTTP/1.1\" 504 132"} {"timestamp": "2023-06-07T10:46:03.267699Z", "level": "DEBUG", "name": "urllib3.util.retry", "message": "Incremented Retry for (url='/api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2'): Retry(total=3, connect=None, read=None, redirect=None, status=None)"} {"timestamp": "2023-06-07T10:46:05.269989Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Retry: /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2"} {"timestamp": "2023-06-07T10:46:05.412953Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "https://otx.alienvault.com:443 \"GET /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2 HTTP/1.1\" 504 132"} {"timestamp": "2023-06-07T10:46:05.413480Z", "level": "DEBUG", "name": "urllib3.util.retry", "message": "Incremented Retry for (url='/api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2'): Retry(total=2, connect=None, read=None, redirect=None, status=None)"} {"timestamp": "2023-06-07T10:46:09.418062Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Retry: /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2"} {"timestamp": "2023-06-07T10:46:09.539907Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "https://otx.alienvault.com:443 \"GET /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2 HTTP/1.1\" 504 132"} {"timestamp": "2023-06-07T10:46:09.540440Z", "level": "DEBUG", "name": "urllib3.util.retry", "message": "Incremented Retry for (url='/api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2'): Retry(total=1, connect=None, read=None, redirect=None, status=None)"} {"timestamp": "2023-06-07T10:46:17.558000Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Retry: /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2"} {"timestamp": "2023-06-07T10:46:23.223383Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Resetting dropped connection: opencti"} {"timestamp": "2023-06-07T10:46:24.398519Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://opencti:8080 \"POST /graphql HTTP/1.1\" 200 98"} {"timestamp": "2023-06-07T10:47:04.415441Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Resetting dropped connection: opencti"} {"timestamp": "2023-06-07T10:47:05.892671Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://opencti:8080 \"POST /graphql HTTP/1.1\" 200 98"} {"timestamp": "2023-06-07T10:47:17.872530Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "https://otx.alienvault.com:443 \"GET /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2 HTTP/1.1\" 504 132"} {"timestamp": "2023-06-07T10:47:17.873202Z", "level": "DEBUG", "name": "urllib3.util.retry", "message": "Incremented Retry for (url='/api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2'): Retry(total=0, connect=None, read=None, redirect=None, status=None)"} {"timestamp": "2023-06-07T10:47:33.886005Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Retry: /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2"} {"timestamp": "2023-06-07T10:47:45.896673Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Resetting dropped connection: opencti"} {"timestamp": "2023-06-07T10:47:46.652088Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://opencti:8080 \"POST /graphql HTTP/1.1\" 200 98"} {"timestamp": "2023-06-07T10:48:26.656244Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Resetting dropped connection: opencti"} {"timestamp": "2023-06-07T10:48:28.011277Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://opencti:8080 \"POST /graphql HTTP/1.1\" 200 98"} {"timestamp": "2023-06-07T10:48:34.149058Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "https://otx.alienvault.com:443 \"GET /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2 HTTP/1.1\" 504 132"} Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 489, in send resp = conn.urlopen( File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 878, in urlopen return self.urlopen( File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 878, in urlopen return self.urlopen( File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 878, in urlopen return self.urlopen( [Previous line repeated 2 more times] File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 868, in urlopen retries = retries.increment(method, url, response=response, _pool=self) File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2 (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/OTXv2.py", line 178, in get response = self.session().get( File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 600, in get return self.request("GET", url, kwargs) File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request resp = self.send(prep, send_kwargs) File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 556, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2 (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/opencti-connector-alienvault/main.py", line 8, in connector.run() File "/opt/opencti-connector-alienvault/alienvault/core.py", line 255, in run pulse_import_state = self.pulse_importer.run(current_state, work_id) File "/opt/opencti-connector-alienvault/alienvault/importer.py", line 100, in run pulses = self._fetch_subscribed_pulses(latest_pulse_datetime) File "/opt/opencti-connector-alienvault/alienvault/importer.py", line 191, in _fetch_subscribed_pulses pulses = self.client.get_pulses_subscribed(modified_since) File "/opt/opencti-connector-alienvault/alienvault/client.py", line 40, in get_pulses_subscribed pulse_data = self.otx.getsince(timestamp=modifie d_since, limit=limit) File "/usr/local/lib/python3.10/site-packages/OTXv2.py", line 431, in getsince return self.getall(limit=limit, modified_since=timestamp, max_page=max_page, max_items=max_items, iter=False) File "/usr/local/lib/python3.10/site-packages/OTXv2.py", line 404, in getall return self.walkapi( File "/usr/local/lib/python3.10/site-packages/OTXv2.py", line 385, in walkapi return list(self.walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body)) File "/usr/local/lib/python3.10/site-packages/OTXv2.py", line 366, in walkapi_iter data = self.get(next_page_url) File "/usr/local/lib/python3.10/site-packages/OTXv2.py", line 187, in get raise RetryError() OTXv2.RetryError: 'Exceeded maximum number of retries' Terminated

Screenshots

connector

FormindMPO commented 1 year ago

Have you ever tried with a smaller ALIENVAULT_PULSE_START_TIMESTAMP value ? Maybe Otx is not able to pull such a big amount of pulses, try with 2023-06-01T00:00:00 ... ?

iman006 commented 1 year ago

Have you ever tried with a smaller ALIENVAULT_PULSE_START_TIMESTAMP value ? Maybe Otx is not able to pull such a big amount of pulses, try with 2023-06-01T00:00:00 ... ?

Thanks for your response. Yes, I tried this as well but it didn't work.

iman006 commented 1 year ago

@richard-julien Do you have any ideas about this problem? I would appreciate it if you could help. :(

richard-julien commented 1 year ago

Looks like AlienVault is responding with 504 errors

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2 (Caused by ResponseError('too many 504 error responses'))

Maybe some rate limiting on alienvault API?

iman006 commented 1 year ago

Looks like AlienVault is responding with 504 errors

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=20&modified_since=2022-08-01T00%3A00%3A00&page=2 (Caused by ResponseError('too many 504 error responses'))

Maybe some rate limiting on alienvault API?

I have tested this connector with various accounts and APIs, but the problem still persists.