[Worker] Error pinging the API

[dominictory]

Hi all,

After upgrading to 6.3.1 from 6.2.18, my workers are constantly generating the below error. It is causing extremely slow ingestion.

{"reason": "HTTPSConnectionPool(host='opencti', port=443): Max retries exceeded with url: /graphql (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))", "headers": "{'User-Agent': 'pycti/6.3.1', 'Authorization': 'Bearer 4e95798b-21a6-4571-819f-d974f2c2abea'}"}

{"timestamp": "2024-09-25T07:57:19.530266Z", "level": "ERROR", "name": "worker", "message": "Error pinging the API", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 466, in _make_request\n self._validate_conn(conn)\n File \"/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 1095, in _validate_conn\n conn.connect()\n File \"/usr/local/lib/python3.12/site-packages/urllib3/connection.py\", line 730, in connect\n sock_and_verified = _ssl_wrap_socket_and_match_hostname(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/urllib3/connection.py\", line 909, in _ssl_wrap_socket_and_match_hostname\n ssl_sock = ssl_wrap_socket(\n ^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py\", line 469, in ssl_wrap_socket\n ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py\", line 513, in _ssl_wrap_socket_impl\n return ssl_context.wrap_socket(sock, server_hostname=server_hostname)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/ssl.py\", line 455, in wrap_socket\n return self.sslsocket_class._create(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/ssl.py\", line 1041, in _create\n self.do_handshake()\n File \"/usr/local/lib/python3.12/ssl.py\", line 1319, in do_handshake\n self._sslobj.do_handshake()\nssl.SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 789, in urlopen\n response = self._make_request(\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 490, in _make_request\n raise new_e\nurllib3.exceptions.SSLError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.12/site-packages/requests/adapters.py\", line 667, in send\n resp = conn.urlopen(\n ^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 843, in urlopen\n retries = retries.increment(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py\", line 519, in increment\n raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='opencti', port=443): Max retries exceeded with url: /graphql (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/opencti-worker/worker.py\", line 58, in ping\n self.api.query(\n File \"/usr/local/lib/python3.12/site-packages/pycti/api/opencti_api_client.py\", line 337, in query\n r = self.session.post(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/requests/sessions.py\", line 637, in post\n return self.request(\"POST\", url, data=data, json=json, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, **send_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/requests/adapters.py\", line 698, in send\n raise SSLError(e, request=request)\nrequests.exceptions.SSLError: HTTPSConnectionPool(host='opencti', port=443): Max retries exceeded with url: /graphql (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))", "taskName": null, "attributes": {"reason": "HTTPSConnectionPool(host='opencti', port=443): Max retries exceeded with url: /graphql (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))", "headers": "{'User-Agent': 'pycti/6.3.1', 'Authorization': 'Bearer 4e95798b-21a6-4571-819f-d974f2c2abea'}"}}

[romain-filigran]

Hello @dominictory . The problem seems to be with the SSL connection. How did you configure the SSL termination?

[dominictory]

Hello @dominictory . The problem seems to be with the SSL connection. How did you configure the SSL termination?

We use a self-signed certificate for the platform, which is still valid. Not sure about SSL termination, I haven't configured anything like that.

[dominictory]

Hello @dominictory . The problem seems to be with the SSL connection. How did you configure the SSL termination?

Slightly different SSL error log (_ssl.c:2406):

{"timestamp": "2024-09-28T16:56:08.674147Z", "level": "ERROR", "name": "worker", "message": "Error pinging the API", "exc_info": "urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:2406)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.12/site-packages/requests/adapters.py\", line 667, in send\n    resp = conn.urlopen(\n           ^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 843, in urlopen\n    retries = retries.increment(\n              ^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py\", line 519, in increment\n    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='opencti', port=443): Max retries exceeded with url: /graphql (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/opt/opencti-worker/worker.py\", line 58, in ping\n    self.api.query(\n  File \"/usr/local/lib/python3.12/site-packages/pycti/api/opencti_api_client.py\", line 337, in query\n    r = self.session.post(\n        ^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/requests/sessions.py\", line 637, in post\n    return self.request(\"POST\", url, data=data, json=json, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/requests/sessions.py\", line 589, in request\n    resp = self.send(prep, **send_kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/requests/sessions.py\", line 703, in send\n    r = adapter.send(request, **kwargs)\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/requests/adapters.py\", line 698, in send\n    raise SSLError(e, request=request)\nrequests.exceptions.SSLError: HTTPSConnectionPool(host='opencti', port=443): Max retries exceeded with url: /graphql (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))", "taskName": null, "attributes": {"reason": "HTTPSConnectionPool(host='opencti', port=443): Max retries exceeded with url: /graphql (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))", "headers": "{'User-Agent': 'pycti/6.3.1', 'Authorization': 'Bearer 4e95798b-21a6-4571-819f-d974f2c2abea'}"}}