search

canonical / opensearch-operator

OpenSearch operator
Apache License 2.0
13 stars 7 forks source link

[CI] `wait_until` is not waiting for executing status to finish - OpenSearch Dashboard CI is failing #370

Closed phvalguima closed 3 months ago

phvalguima commented 3 months ago

The CI is failing with:

Traceback (most recent call last):
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 241, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 87, in pytest_runtest_call
    yield from thread_exception_runtest_hook()
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
    yield
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 90, in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
    yield
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 850, in pytest_runtest_call
    yield from self._runtest_for(item, "call")
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 833, in _runtest_for
    yield
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/capture.py", line 878, in pytest_runtest_call
    return (yield)
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/skipping.py", line 257, in pytest_runtest_call
    return (yield)
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 183, in pytest_runtest_call
    raise e
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 173, in pytest_runtest_call
    item.runtest()
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 1632, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 162, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/runner/work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pytest_asyncio/plugin.py", line 529, in inner
    _loop.run_until_complete(task)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/runner/work/opensearch-operator/opensearch-operator/tests/integration/relations/test_opensearch_provider.py", line 234, in test_dashboard_relation
    secret_uri = await get_application_relation_data(
  File "/home/runner/work/opensearch-operator/opensearch-operator/tests/integration/relations/helpers.py", line 57, in get_application_relation_data
    raise ValueError(
ValueError: no relation data could be grabbed on relation with endpoint opensearch_client

Looking at the juju status, we can see the units were not yet settled:

INFO     integration.helpers_deployments:helpers_deployments.py:330 Model  Controller           Cloud/Region         Version  SLA          Timestamp
test   localhost-localhost  localhost/localhost  3.4.3    unsupported  11:45:00Z

App                       Version  Status   Scale  Charm                     Channel        Rev  Exposed  Message
application                        active       1  application                                0  no       
opensearch                         active       3  opensearch                                 0  no       
opensearch-dashboards              blocked      1  opensearch-dashboards     2/edge          15  no       Opensearch connection is missing
self-signed-certificates           active       1  self-signed-certificates  latest/stable  155  no       

Unit                         Workload  Agent      Machine  Public address  Ports     Message
application/0*               active    idle       2        10.56.58.252              
opensearch-dashboards/0*     active    executing  1        10.56.58.227    5601/tcp  
opensearch/0                 active    executing  3        10.56.58.87     9200/tcp  
opensearch/1                 active    executing  4        10.56.58.192    9200/tcp  
opensearch/2*                active    idle       5        10.56.58.214    9200/tcp  
self-signed-certificates/0*  active    idle       0        10.56.58.254              

Machine  State    Address       Inst id        Base          AZ  Message
0        started  10.56.58.254  juju-7df6f8-0  ubuntu@22.04      Running
1        started  10.56.58.227  juju-7df6f8-1  ubuntu@22.04      Running
2        started  10.56.58.252  juju-7df6f8-2  ubuntu@22.04      Running
3        started  10.56.58.87   juju-7df6f8-3  ubuntu@22.04      Running
4        started  10.56.58.192  juju-7df6f8-4  ubuntu@22.04      Running
5        started  10.56.58.214  juju-7df6f8-5  ubuntu@22.04      Running

It is unclear as why this behavior started now.

github-actions[bot] commented 3 months ago

https://warthogs.atlassian.net/browse/DPE-4892

phvalguima commented 3 months ago

Fixed with latest commit from Mehdi