FH-CrOSSD / crossd

Critical Open-Source Software Database (CrOSSD)
https://health.crossd.tech
Creative Commons Attribution Share Alike 4.0 International
1 stars 0 forks source link

worker-drone crash when arango unreachable #1

Open tobiasdam opened 7 months ago

tobiasdam commented 7 months ago
[2024-02-08 10:43:42,037: WARNING/MainProcess] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1df02d1950>: Failed to establish a new connection: [Errno 111] Connection refused')': /_api/user/root/database
[2024-02-08 10:43:43,087: WARNING/MainProcess] ===
Unable to establish connection, perhaps arango is not running.
===
[2024-02-08 10:43:43,087: ERROR/MainProcess] Pool callback raised exception: ConnectionError(MaxRetryError("HTTPSConnectionPool(host='arangodb-cluster-internal', port=8529): Max retries exceeded with url: /_api/user/root/database (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1df01bd010>: Failed to establish a new connection: [Errno 111] Connection refused'))"))
Traceback (most recent call last):
  File "/project/pkgs/urllib3/connection.py", line 203, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/pkgs/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/project/pkgs/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/pkgs/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/project/pkgs/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
  File "/project/pkgs/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/project/pkgs/urllib3/connectionpool.py", line 1092, in _validate_conn
    conn.connect()
  File "/project/pkgs/urllib3/connection.py", line 611, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "/project/pkgs/urllib3/connection.py", line 218, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f1df01bd010>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/pkgs/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/project/pkgs/urllib3/connectionpool.py", line 874, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/project/pkgs/urllib3/connectionpool.py", line 874, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/project/pkgs/urllib3/connectionpool.py", line 874, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  [Previous line repeated 2 more times]
  File "/project/pkgs/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/project/pkgs/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='arangodb-cluster-internal', port=8529): Max retries exceeded with url: /_api/user/root/database (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1df01bd010>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/collector-drone/.venv/lib/python3.11/site-packages/billiard/pool.py", line 1793, in safe_apply_callback
    fun(*args, **kwargs)
  File "/home/collector-drone/.venv/lib/python3.11/site-packages/celery/worker/request.py", line 624, in on_failure
    self.task.backend.mark_as_failure(
  File "/home/collector-drone/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 167, in mark_as_failure
    self.store_result(task_id, exc, state,
  File "/home/collector-drone/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 526, in store_result
    self._store_result(task_id, result, state, traceback,
  File "/home/collector-drone/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 973, in _store_result
    current_meta = self._get_task_meta_for(task_id)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/collector-drone/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 995, in _get_task_meta_for
    meta = self.get(self.get_key_for_task(task_id))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/collector-drone/.venv/lib/python3.11/site-packages/celery/backends/arangodb.py", line 123, in get
    query = self.db.AQLQuery(
            ^^^^^^^
  File "/home/collector-drone/.venv/lib/python3.11/site-packages/celery/backends/arangodb.py", line 114, in db
    return self.connection[self.database]
           ^^^^^^^^^^^^^^^
  File "/home/collector-drone/.venv/lib/python3.11/site-packages/celery/backends/arangodb.py", line 105, in connection
    self._connection = py_arango_connection.Connection(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/collector-drone/.venv/lib/python3.11/site-packages/pyArango/connection.py", line 244, in __init__
    self.reload()
  File "/home/collector-drone/.venv/lib/python3.11/site-packages/pyArango/connection.py", line 338, in reload
    r = self.session.get(self.getDatabasesURL())
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/collector-drone/.venv/lib/python3.11/site-packages/pyArango/connection.py", line 57, in __call__
    ret = self.fct(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/pkgs/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/pkgs/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/pkgs/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/pkgs/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='arangodb-cluster-internal', port=8529): Max retries exceeded with url: /_api/user/root/database (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1df01bd010>: Failed to establish a new connection: [Errno 111] Connection refused'))