search

juju / python-libjuju

Python library for the Juju API
Apache License 2.0
59 stars 97 forks source link

Improve the usefulness of JujuUnitError #972

Open sed-i opened 10 months ago

sed-i commented 10 months ago

Description

Quite often we have itests that fail in CI with insufficient context. For example:

______________________________ test_deploy_charms ______________________________
Traceback (most recent call last):
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 177, in pytest_runtest_call
    raise e
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
    item.runtest()
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 1792, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/pytest_asyncio/plugin.py", line 532, in inner
    _loop.run_until_complete(task)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/tests/integration/test_upgrade_charm_retains_alerts.py", line 49, in test_deploy_charms
    await ops_test.model.wait_for_idle(apps=[app_name, rules_app], status="active", timeout=1000)
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/juju/model.py", line 2751, in wait_for_idle
    _raise_for_status(errors, "error")
  File "/home/runner/work/loki-k8s-operator/loki-k8s-operator/.tox/integration/lib/python3.10/site-packages/juju/model.py", line 2657, in _raise_for_status
    raise error_type("{}{} in {}: {}".format(
juju.errors.JujuUnitError: Unit in error: loki-tester/0

The message juju.errors.JujuUnitError: Unit in error: loki-tester/0 doesn't help much.

Urgency

Need this really badly for test suite

Code I'd Like to Run

It would be awesome if the JujuUnitError included more context, in particular the status message of the unit (perhaps it includes the reason).

lucasgameiroborges commented 2 months ago

Just wanted to chime in here as this issue really affects the clarity of errors in our CI for Database Charms. Also, this holds true for all types of error that wait_for_idle can raise, for example JujuAppError. We need, at the very least, the error message being raised. Example below:

Traceback (most recent call last):
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 241, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 87, in pytest_runtest_call
    yield from thread_exception_runtest_hook()
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
    yield
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 90, in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
    yield
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 850, in pytest_runtest_call
    yield from self._runtest_for(item, "call")
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 833, in _runtest_for
    yield
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/capture.py", line 878, in pytest_runtest_call
    return (yield)
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/skipping.py", line 257, in pytest_runtest_call
    return (yield)
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 183, in pytest_runtest_call
    raise e
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 173, in pytest_runtest_call
    item.runtest()
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 1632, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 162, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/pytest_asyncio/plugin.py", line 529, in inner
    _loop.run_until_complete(task)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/tests/integration/ha_tests/test_self_healing.py", line 78, in test_build_and_deploy
    await ops_test.model.wait_for_idle(status="active", timeout=1000)
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/juju/model.py", line 2623, in wait_for_idle
    _raise_for_status(errors, "error")
  File "/home/runner/work/postgresql-k8s-operator/postgresql-k8s-operator/.tox/integration/lib/python3.10/site-packages/juju/model.py", line 2535, in _raise_for_status
    raise error_type("*** in ***: ***".format(
juju.errors.JujuAppError: App in error: postgresql-k8s
taurus-forever commented 1 month ago

we are experiencing the same App in error problem randomly, e.g. here for mysql-k8s.

Extra information is in Juju app status message: create Pod db1-1 in StatefulSet db1 failed error: failed to create PVC db1-database-569a5ebe-db1-1: persistentvolumec...

Searching a good hint to continue from here. Tnx!

taurus-forever commented 4 weeks ago

The issue is random but popular: pgbouncer-k8s.

How to collect extra debug here?