search

canonical / data-platform-workflows

Reusable GitHub Actions workflows used by the Data Platform team
Apache License 2.0
3 stars 9 forks source link

[self-hosted] Microceph setup fixture often fails with: "error to connect to http://localhost/testbucket" #171

Closed phvalguima closed 2 months ago

phvalguima commented 2 months ago

We have two types of backup tests: 2x runners as self-hosted (bigger VMs) to run the large-deployments; and 2x runners as GH-hosted for the small-deployments in OpenSearch.

The self-hosted runners are often failing in the microceph setup. One example: https://github.com/canonical/opensearch-operator/actions/runs/8959102660/job/24604651827

The error rate is often enough (last 2x runs I executed, 3 of 4 tests for backup with self-hosted failed with this error).

Given this error is only observed on self-hosted, I believe it has to do with the NO_PROXY settings. Generally, we must have in our no-proxy setup something like:

# If we already have some content on NO_PROXY
export NO_PROXY=$NO_PROXY,::1,127.0.0.1,localhost

This ensures the host will not forward local-bound traffic to the proxy and instead resolve within the host.

The full stack trace is:

Traceback (most recent call last):
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/httpsession.py", line 464, in send
    urllib_response = conn.urlopen(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/awsrequest.py", line 96, in request
    rval = super().request(method, url, body, headers, *args, **kwargs)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/awsrequest.py", line 123, in _send_output
    self.send(msg)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/awsrequest.py", line 223, in send
    return super().send(str)
  File "/usr/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPConnection object at 0x71360e1e78e0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 241, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 85, in pytest_runtest_setup
    yield from unraisable_exception_runtest_hook()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 844, in pytest_runtest_setup
    yield from self._runtest_for(item, "setup")
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 833, in _runtest_for
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/capture.py", line 873, in pytest_runtest_setup
    return (yield)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 82, in pytest_runtest_setup
    yield from thread_exception_runtest_hook()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 159, in pytest_runtest_setup
    item.session._setupstate.setup(item)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 515, in setup
    raise exc
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 512, in setup
    col.setup()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 1630, in setup
    self._request._fillfixtures()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 683, in _fillfixtures
    item.funcargs[argname] = self.getfixturevalue(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 518, in getfixturevalue
    fixturedef = self._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1035, in execute
    fixturedef = request._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1035, in execute
    fixturedef = request._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1053, in execute
    raise exc
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 241, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 85, in pytest_runtest_setup
    yield from unraisable_exception_runtest_hook()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 844, in pytest_runtest_setup
    yield from self._runtest_for(item, "setup")
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 833, in _runtest_for
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/capture.py", line 873, in pytest_runtest_setup
    return (yield)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 82, in pytest_runtest_setup
    yield from thread_exception_runtest_hook()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 159, in pytest_runtest_setup
    item.session._setupstate.setup(item)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 515, in setup
    raise exc
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 512, in setup
    col.setup()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 1630, in setup
    self._request._fillfixtures()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 683, in _fillfixtures
    item.funcargs[argname] = self.getfixturevalue(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 518, in getfixturevalue
    fixturedef = self._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1035, in execute
    fixturedef = request._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1035, in execute
    fixturedef = request._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1053, in execute
    raise exc
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 241, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 85, in pytest_runtest_setup
    yield from unraisable_exception_runtest_hook()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 844, in pytest_runtest_setup
    yield from self._runtest_for(item, "setup")
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 833, in _runtest_for
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/capture.py", line 873, in pytest_runtest_setup
    return (yield)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 82, in pytest_runtest_setup
    yield from thread_exception_runtest_hook()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 159, in pytest_runtest_setup
    item.session._setupstate.setup(item)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 515, in setup
    raise exc
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 512, in setup
    col.setup()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 1630, in setup
    self._request._fillfixtures()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 683, in _fillfixtures
    item.funcargs[argname] = self.getfixturevalue(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 518, in getfixturevalue
    fixturedef = self._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1035, in execute
    fixturedef = request._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1035, in execute
    fixturedef = request._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1073, in execute
    result = ihook.pytest_fixture_setup(fixturedef=self, request=request)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 167, in _multicall
    teardown.throw(outcome._exception)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup
    return (yield)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1122, in pytest_fixture_setup
    result = call_fixture_func(fixturefunc, request, kwargs)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 884, in call_fixture_func
    fixture_result = fixturefunc(**kwargs)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pytest_microceph/_plugin.py", line 46, in microceph
    boto3.client(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/client.py", line [565](https://github.com/canonical/opensearch-operator/actions/runs/8959102660/job/24604651827#step:22:566), in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/client.py", line 1001, in _make_api_call
    http, parsed_response = self._make_request(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/client.py", line 1027, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/endpoint.py", line 202, in _send_request
    while self._needs_retry(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/endpoint.py", line 354, in _needs_retry
    responses = self._event_emitter.emit(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/retryhandler.py", line 207, in __call__
    if self._checker(**checker_kwargs):
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/retryhandler.py", line 284, in __call__
    should_retry = self._should_retry(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/retryhandler.py", line 320, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/retryhandler.py", line 363, in __call__
    checker_response = checker(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/retryhandler.py", line 247, in __call__
    return self._check_caught_exception(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
    raise caught_exception
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/endpoint.py", line 281, in _do_get_response
    http_response = self._send(request)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/endpoint.py", line 377, in _send
    return self.http_session.send(request)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/httpsession.py", line 493, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost/testbucket"
github-actions[bot] commented 2 months ago

https://warthogs.atlassian.net/browse/DPE-4289

carlcsaposs-canonical commented 2 months ago

Posting in https://chat.canonical.com/canonical/channels/github-actions-self-hosted-runners, since it appears to be an issue with self-hosted runners

I believe aproxy is supposed to be configured on the self-hosted runners so that no changes are needed on our end with regards to the proxy

carlcsaposs-canonical commented 2 months ago

fyi: https://github.com/canonical/data-platform-workflows/blob/main/.github/workflows/integration_test_charm.md#user-content-fn-1-bccca2791f328953660bf6ef3bd00a3e