kernelci / kernelci-pipeline

Modular pipeline based on the KernelCI API
GNU Lesser General Public License v2.1
6 stars 18 forks source link

'Max retries exceeded' error when running result_summary #560

Closed nfraprado closed 4 months ago

nfraprado commented 4 months ago

Just tried running the result summary on the latest commit like so:

docker-compose run result_summary --preset=mainline-next-regressions

But got a traceback:

04/23/2024 09:57:39 PM UTC [DEBUG] Query: {'kind': 'regression', 'state': 'done', 'data.error_code': 'null', 'data.failed_kernel_version.tree': 'mainline', 'created__gt': '2024-04-22T21:57:39', 'created__lt': '2024-04-23T21:57:39'}
04/23/2024 09:57:39 PM UTC [INFO] Searching
04/23/2024 09:57:39 PM UTC [ERROR] Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 400, in request
    self.endheaders()
  File "/usr/local/lib/python3.11/http/client.py", line 1289, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1048, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.11/http/client.py", line 986, in send
    self.connect()
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 238, in connect
    self.sock = self._new_conn()
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 213, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x71992298bd90>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.17.0.1', port=8001): Max retries exceeded with url: /latest/nodes?kind=regression&state=done&data.error_code=null&data.failed_kernel_version.tree=mainline&created__gt=2024-04-22T21%3A57%3A39&created__lt=2024-04-23T21%3A57%3A39&limit=100 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x71992298bd90>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kernelci/pipeline/base.py", line 69, in run
    status = self._run(context)
             ^^^^^^^^^^^^^^^^^^
  File "/home/kernelci/./pipeline/result_summary.py", line 140, in _run
    summary.run(self, context)
  File "/home/kernelci/pipeline/result_summary/summary.py", line 65, in run
    query_results = utils.iterate_node_find(service, params_set)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kernelci/pipeline/result_summary/utils.py", line 75, in iterate_node_find
    search = service._api.node.find(params, limit=limit, offset=offset)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kernelci/api/latest.py", line 114, in find
    return self._get_paginated(params, 'nodes', offset, limit)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kernelci/api/__init__.py", line 147, in _get_paginated
    resp = self._get(path, params=params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kernelci/api/__init__.py", line 66, in _get
    resp = requests.get(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.17.0.1', port=8001): Max retries exceeded with url: /latest/nodes?kind=regression&state=done&data.error_code=null&data.failed_kernel_version.tree=mainline&created__gt=2024-04-22T21%3A57%3A39&created__lt=2024-04-23T21%3A57%3A39&limit=100 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x71992298bd90>: Failed to establish a new connection: [Errno 111] Connection refused'))

@hardboprobot any idea what's causing this? This wasn't an issue on e2bcc7dc6ae2, which I was using before checking out the latest commit.

r-c-n commented 4 months ago

It looks like it's trying to connect to a local docker-based KernelCI instance instead of connecting to the staging instance. Can you check that api_config points to "staging" in your config/kernelci.toml file?

nfraprado commented 4 months ago

Yep, that did it, I forgot to re-do that when checking out the latest commit, my bad. Thank you for the help :).