search

charmed-kubernetes / actions-operator

Apache License 2.0
7 stars 18 forks source link

Kubernetes model doesn't clean up #3

Closed chris-sanders closed 3 years ago

chris-sanders commented 3 years ago

Running this on top of a Kubernetes cluster with a charm, I found that when the charm failed on the config hook the model isn't cleaned up. At first glance this might be a difference in how 'machines' are returned on a container model.

Here's the output showing where it failed. It's a little verbose but shows the error.

$ tox -e integration
pyenv doesn't seem to be installed, you probably don't want this plugin installed either.
integration installed: appdirs==1.4.4,attrs==20.3.0,backcall==0.2.0,bcrypt==3.2.0,blessings==1.6,certifi==2020.12.5,cffi==1.14.5,chardet==3.0.4,charm-tools==2.8.3,charmcraft==0.7.0,Cheetah3==3.2.6.post1,colander==1.7.0,cryptography==3.4.6,decorator==4.4.2,dict2colander==0.2,distlib==0.3.1,distro==1.5.0,filelock==3.0.12,httplib2==0.19.0,idna==2.10,iniconfig==1.1.1,ipdb==0.13.6,ipython==7.21.0,ipython-genutils==0.2.0,iso8601==0.1.14,jedi==0.18.0,Jinja2==2.11.2,jsonschema==2.5.1,juju @ https://github.com/juju/python-libjuju/archive/master.zip,jujubundlelib==0.5.6,keyring==20.0.1,launchpadlib==1.10.13,lazr.restfulclient==0.14.3,lazr.uri==1.0.5,libcharmstore==0.0.9,macaroonbakery==1.3.1,MarkupSafe==1.1.1,mypy-extensions==0.4.3,oauthlib==3.1.0,otherstuf==1.1.0,packaging==20.9,paramiko==2.7.2,parse==1.19.0,parso==0.8.1,path==15.1.2,path.py==12.5.0,pathspec==0.3.4,pbr==5.5.1,pexpect==4.8.0,pickleshare==0.7.5,pluggy==0.13.1,prompt-toolkit==3.0.16,protobuf==3.15.5,ptyprocess==0.7.0,py==1.10.0,pyasn1==0.4.8,pycparser==2.20,Pygments==2.8.1,pymacaroons==0.13.0,PyNaCl==1.4.0,pyparsing==2.4.7,pyRFC3339==1.1,pytest==6.2.2,pytest-operator==0.5.1,python-dateutil==2.8.1,pytz==2021.1,PyYAML==5.3.1,requests==2.24.0,requests-toolbelt==0.9.1,requirements-parser==0.2.0,ruamel.yaml==0.15.100,SecretStorage==2.3.1,six==1.15.0,stuf==0.9.16,tabulate==0.8.7,testresources==2.0.1,theblues==0.5.2,toml==0.10.2,toposort==1.6,traitlets==5.0.5,translationstring==1.4,typing-extensions==3.7.4.3,typing-inspect==0.6.0,urllib3==1.25.11,vergit==1.0.2,virtualenv==20.4.2,wadllib==1.3.5,wcwidth==0.2.5,websockets==7.0
integration run-test-pre: PYTHONHASHSEED='4220337346'
integration run-test: commands[0] | pytest -v --tb native --show-capture=no --log-cli-level=INFO -s /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/tests/integration
==================================================================== test session starts =====================================================================
platform linux -- Python 3.8.5, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/bin/python
cachedir: .tox/integration/.pytest_cache
rootdir: /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator
plugins: operator-0.5.1
collected 2 items

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
----------------------------------------------------------------------- live log setup -----------------------------------------------------------------------
INFO     pytest_operator.plugin:plugin.py:123 Using tmp_path: /tmp/pytest-of-chris/pytest-42/integration-tests-ytqc0
INFO     pytest_operator.plugin:plugin.py:196 Adding model micro:integration-tests-zh8f
----------------------------------------------------------------------- live log call ------------------------------------------------------------------------
INFO     pytest_operator.plugin:plugin.py:313 Building charm gatekeeper-audit
INFO     juju.model:model.py:1546 Deploying local:kubernetes/gatekeeper-audit-0
INFO     juju.model:model.py:2257 Waiting for model:
  kubernetes/0 [allocating] waiting: installing agent
INFO     juju.model:model.py:2257 Waiting for model:
  kubernetes/0 [allocating] waiting: agent initializing
FAILED
tests/integration/test_charm.py::IntegrationTests::test_status_messages XFAIL (aborted)
tests/integration/test_charm.py::IntegrationTests::test_status_messages ERROR

=========================================================================== ERRORS ===========================================================================
_________________________________________________ ERROR at teardown of IntegrationTests.test_status_messages _________________________________________________
Traceback (most recent call last):
  File "/home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/pytest_operator/plugin.py", line 132, in inject_fixtures
    cls.loop.run_until_complete(cls.cleanup_model())
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/pytest_operator/plugin.py", line 262, in cleanup_model
    await cls.dump_model()
  File "/home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/pytest_operator/plugin.py", line 223, in dump_model
    unit.machine.id,
AttributeError: 'NoneType' object has no attribute 'id'
========================================================================== FAILURES ==========================================================================
___________________________________________________________ IntegrationTests.test_build_and_deploy ___________________________________________________________
Traceback (most recent call last):
  File "/home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/tests/integration/test_charm.py", line 21, in test_build_and_deploy
    await self.model.wait_for_idle(wait_for_active=True, timeout=60 * 60)
  File "/home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/juju/model.py", line 2249, in wait_for_idle
    _raise_for_status(errors, "error")
  File "/home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/juju/model.py", line 2201, in _raise_for_status
    raise error_type("{}{} in {}: {}".format(
juju.errors.JujuUnitError: Unit in error: kubernetes/0
====================================================================== warnings summary ======================================================================
tests/integration/test_charm.py:15
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/tests/integration/test_charm.py:15: PytestUnknownMarkWarning: Unknown pytest.mark.order - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.order("first")

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py:783: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    self.stopped = asyncio.Event(loop=loop)

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py:166: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    self.reconnecting = asyncio.Lock(loop=connection.loop)

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py:167: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    self.close_called = asyncio.Event(loop=connection.loop)

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
  /usr/lib/python3.8/asyncio/tasks.py:578: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    done = Queue(loop=loop)

tests/integration/test_charm.py: 45 warnings
  /usr/lib/python3.8/asyncio/queues.py:48: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    self._finished = locks.Event(loop=loop)

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py:614: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    for task in asyncio.as_completed(tasks, loop=self.loop):

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:218: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    self._drain_lock = asyncio.Lock(loop=loop)

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:977: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.sleep(self.ping_interval, loop=self.loop)

tests/integration/test_charm.py: 46 warnings
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:911: DeprecationWarning: 'with (yield from lock)' is deprecated use 'async with lock' instead
    with (yield from self._drain_lock):

tests/integration/test_charm.py: 42 warnings
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/juju/utils.py:66: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    value = await self._queues[id].get()

tests/integration/test_charm.py: 74 warnings
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/juju/utils.py:127: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    done, pending = await asyncio.wait([task] + event_tasks,

tests/integration/test_charm.py: 35 warnings
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:416: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait(

tests/integration/test_charm.py: 16 warnings
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py:411: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    await asyncio.sleep(10, loop=self.loop)

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/juju/model.py:461: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    self._watch_stopping = asyncio.Event(loop=self._connector.loop)

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/juju/model.py:462: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    self._watch_stopped = asyncio.Event(loop=self._connector.loop)

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/juju/model.py:463: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    self._watch_received = asyncio.Event(loop=self._connector.loop)

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:532: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:554: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:1077: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:988: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy
  /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/lib/python3.8/site-packages/juju/model.py:992: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    q = asyncio.Queue(loop=self._connector.loop)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================================================================== short test summary info ===================================================================
FAILED tests/integration/test_charm.py::IntegrationTests::test_build_and_deploy - juju.errors.JujuUnitError: Unit in error: kubernetes/0
ERROR tests/integration/test_charm.py::IntegrationTests::test_status_messages - AttributeError: 'NoneType' object has no attribute 'id'
=============================================== 1 failed, 1 xfailed, 288 warnings, 1 error in 65.86s (0:01:05) ===============================================
ERROR: InvocationError for command /home/chris/src/charmed-kubernetes/k8s-opa-operator/opa-audit-operator/.tox/integration/bin/pytest -v --tb native --show-capture=no --log-cli-level=INFO -s tests/integration (exited with code 1)
__________________________________________________________________________ summary ___________________________________________________________________________
ERROR:   integration: commands failed
chris-sanders commented 3 years ago

This is actually on the pytest-operator not the action, closing here.