search

canonical / opensearch-operator

OpenSearch operator
Apache License 2.0
12 stars 7 forks source link

[CI][test_plugins.py] `build_and_deploy` finishes too early, before grafana-agent is done setting itself up #475

Closed phvalguima closed 1 month ago

phvalguima commented 1 month ago

Some CI runs are failing in test_plugins.py in test_monitoring_user_fetch_prometheus_data. CI run example

Error: 

AssertionError: assert 0 > 500
 +  where 0 = <built-in method count of str object at 0x79c198ced030>('opensearch_')
 +    where <built-in method count of str object at 0x79c198ced030> = 'Unauthorized'.count

Status:

Model  Controller           Cloud/Region         Version  SLA          Timestamp
test   localhost-localhost  localhost/localhost  3.[5](https://github.com/canonical/opensearch-operator/actions/runs/11310272526/job/31455221638#step:37:6).3    unsupported  03:32:11Z

App                       Version  Status   Scale  Charm                     Channel        Rev  Exposed  Message
failover                           active       2  opensearch                                 0  no       
grafana-agent                      blocked      4  grafana-agent             latest/edge    28[6](https://github.com/canonical/opensearch-operator/actions/runs/11310272526/job/31455221638#step:37:7)  no       Missing ['grafana-cloud-config']|['grafana-dashboards-provider']|['logging-consumer']|['send-remote-write'] for cos-a...
main                               active       1  opensearch                                 1  no       
opensearch                         active       1  opensearch                                 2  no       
self-signed-certificates           active       1  self-signed-certificates  latest/stable  155  no       

Unit                         Workload     Agent      Machine  Public address  Ports     Message
failover/0                   active       idle       3        10.64.150.109   9200/tcp  
  grafana-agent/3            maintenance  executing           10.64.150.109             (install) Installing grafana-agent snap
failover/1*                  active       idle       4        10.64.150.206   9200/tcp  
  grafana-agent/1*           maintenance  executing           10.64.150.206             (install) Installing grafana-agent snap
main/0*                      active       idle       1        10.64.150.1[8](https://github.com/canonical/opensearch-operator/actions/runs/11310272526/job/31455221638#step:37:9)5   9200/tcp  
  grafana-agent/2            blocked      idle                10.64.150.185             Missing ['grafana-cloud-config']|['grafana-dashboards-provider']|['logging-consumer']|['send-remote-write'] for cos-a...
opensearch/0*                active       idle       2        10.64.150.200   [9](https://github.com/canonical/opensearch-operator/actions/runs/11310272526/job/31455221638#step:37:10)200/tcp  
  grafana-agent/4            blocked      idle                10.64.150.200             Missing ['grafana-cloud-config']|['grafana-dashboards-provider']|['logging-consumer']|['send-remote-write'] for cos-a...
self-signed-certificates/0*  active       idle       0        [10](https://github.com/canonical/opensearch-operator/actions/runs/11310272526/job/31455221638#step:37:11).64.150.154             

Machine  State    Address        Inst id        Base          AZ  Message
0        started  10.64.150.154  juju-9c62cd-0  ubuntu@22.04      Running
1        started  10.64.150.185  juju-9c62cd-1  ubuntu@22.04      Running
2        started  10.64.[15](https://github.com/canonical/opensearch-operator/actions/runs/11310272526/job/31455221638#step:37:16)0.200  juju-9c62cd-2  ubuntu@22.04      Running
3        started  10.64.150.109  juju-9c62cd-3  ubuntu@22.04      Running
4        started  10.64.150.206  juju-9c62cd-4  ubuntu@22.04      Running

The wait_until here must also wait for the grafana-agent.

Full error stack:

Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 341, in from_call
    result: TResult | None = func()
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 242, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 167, in _multicall
    teardown.throw(outcome._exception)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 92, in pytest_runtest_call
    yield from thread_exception_runtest_hook()
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
    yield
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 167, in _multicall
    teardown.throw(outcome._exception)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 95, in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
    yield
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 167, in _multicall
    teardown.throw(outcome._exception)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 846, in pytest_runtest_call
    yield from self._runtest_for(item, "call")
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 829, in _runtest_for
    yield
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 167, in _multicall
    teardown.throw(outcome._exception)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/capture.py", line 880, in pytest_runtest_call
    return (yield)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 167, in _multicall
    teardown.throw(outcome._exception)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/skipping.py", line 257, in pytest_runtest_call
    return (yield)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
    item.runtest()
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 1627, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 159, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pytest_asyncio/plugin.py", line 529, in inner
    _loop.run_until_complete(task)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/ubuntu/actions-runner/_work/opensearch-operator/opensearch-operator/tests/integration/plugins/test_plugins.py", line 304, in test_monitoring_user_fetch_prometheus_data
    assert response_str.count("opensearch_") > 500
AssertionError: assert 0 > 500
 +  where 0 = <built-in method count of str object at 0x79c198ced030>('opensearch_')
 +    where <built-in method count of str object at 0x79c198ced030> = 'Unauthorized'.count
syncronize-issues-to-jira[bot] commented 1 month ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-5666.

This message was autogenerated