Thanks for creating and maintaining the pytest operator, it makes integration tests much easier! This is a fairly broad feature request (or potentially a bug report). Recently I have been testing a charm on kubernetes that depends on another charm that can only run on lxd. The first thing I tried was to have a controller running on lxd and use add-k8s to also get the lxd controller to manage the kubernetes charm. Then I used the --cloud and --controller parameters to request pytest operator to use the lxd controller and kubernetes cloud. This resulted in the following error (I also tried this on GitHub actions with the same result, it looks like some weird unauthorized error):
tox -e integration -- --jenkins-agent-image localhost:32000/jenkins-agent-k8s:85f19a452481f845507e9457796022a751aa039b --cloud micro-lxd --controller lxd --jenkins-controller-name lxd --jenkins-model-name jenkins --jenkins-unit-number 1
integration installed: asttokens==2.0.8,attrs==22.1.0,backcall==0.2.0,bcrypt==3.2.2,cachetools==5.2.0,certifi==2022.6.15,cffi==1.15.1,charset-normalizer==2.1.0,cryptography==37.0.4,decorator==5.1.1,executing==0.10.0,google-auth==2.10.0,idna==3.3,iniconfig==1.1.1,ipdb==0.13.9,ipython==8.4.0,jedi==0.18.1,Jinja2==3.1.2,juju==3.0.1,jujubundlelib==0.5.7,kubernetes==24.2.0,macaroonbakery==1.3.1,MarkupSafe==2.1.1,matplotlib-inline==0.1.6,multi-key-dict==2.0.3,mypy-extensions==0.4.3,oauthlib==3.2.0,ops==1.5.2,packaging==21.3,paramiko==2.11.0,parso==0.8.3,pbr==5.10.0,pexpect==4.8.0,pickleshare==0.7.5,pluggy==1.0.0,prompt-toolkit==3.0.30,protobuf==3.20.1,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pycparser==2.21,Pygments==2.13.0,pymacaroons==0.13.0,PyNaCl==1.5.0,pyparsing==3.0.9,pyRFC3339==1.1,pytest==7.1.2,pytest-asyncio==0.19.0,pytest-operator==0.22.0,python-dateutil==2.8.2,python-jenkins==1.7.0,pytz==2022.2.1,PyYAML==6.0,requests==2.28.1,requests-oauthlib==1.3.1,rsa==4.9,six==1.16.0,stack-data==0.4.0,theblues==0.5.2,toml==0.10.2,tomli==2.0.1,toposort==1.7,traitlets==5.3.0,typing-inspect==0.8.0,typing_extensions==4.3.0,urllib3==1.26.11,wcwidth==0.2.5,websocket-client==1.3.3,websockets==10.3
integration run-test-pre: PYTHONHASHSEED='2230270510'
integration run-test: commands[0] | pytest -v --tb native --ignore=/home/jdkandersson/src/jenkins-agent-operator/tests/unit --log-cli-level=INFO -s --jenkins-agent-image localhost:32000/jenkins-agent-k8s:85f19a452481f845507e9457796022a751aa039b --cloud micro-lxd --controller lxd --jenkins-controller-name lxd --jenkins-model-name jenkins --jenkins-unit-number 1
===================================================================================================== test session starts =====================================================================================================
platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 -- /home/jdkandersson/src/jenkins-agent-operator/.tox/integration/bin/python
cachedir: .tox/integration/.pytest_cache
rootdir: /home/jdkandersson/src/jenkins-agent-operator, configfile: pyproject.toml
plugins: operator-0.22.0, asyncio-0.19.0
asyncio: mode=strict
collected 2 items
tests/integration/test_charm.py::test_active
------------------------------------------------------------------------------------------------------- live log setup --------------------------------------------------------------------------------------------------------
INFO pytest_operator.plugin:plugin.py:625 Adding model lxd:test-charm-cmhu on cloud micro-lxd
ERROR
tests/integration/test_charm.py::test_build_succeeds ERROR
=========================================================================================================== ERRORS ============================================================================================================
________________________________________________________________________________________________ ERROR at setup of test_active ________________________________________________________________________________________________
Traceback (most recent call last):
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/pytest_asyncio/plugin.py", line 293, in _asyncgen_fixture_wrapper
result = event_loop.run_until_complete(setup())
File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/pytest_asyncio/plugin.py", line 275, in setup
res = await gen_obj.__anext__()
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/pytest_operator/plugin.py", line 231, in ops_test
await ops_test._setup_model()
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/pytest_operator/plugin.py", line 690, in _setup_model
model_state = await self._add_model(
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/pytest_operator/plugin.py", line 627, in _add_model
model = await controller.add_model(model_name, cloud_name, **kwargs)
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/juju/controller.py", line 342, in add_model
model_info = await model_facade.CreateModel(
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/juju/client/facade.py", line 481, in wrapper
reply = await f(*args, **kwargs)
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/juju/client/old_clients/_client9.py", line 3158, in CreateModel
reply = await self.rpc(msg)
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/juju/client/facade.py", line 654, in rpc
result = await self.connection.rpc(msg, encoder=TypeEncoder)
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/juju/client/connection.py", line 634, in rpc
raise errors.JujuAPIError(result)
juju.errors.JujuAPIError: failed to open kubernetes client: unable to determine legacy status for namespace test-charm-cmhu: Unauthorized
----------------------------------------------------------------------------------------------------- Captured log setup ------------------------------------------------------------------------------------------------------
INFO pytest_operator.plugin:plugin.py:625 Adding model lxd:test-charm-cmhu on cloud micro-lxd
____________________________________________________________________________________________ ERROR at setup of test_build_succeeds ____________________________________________________________________________________________
Traceback (most recent call last):
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/pytest_asyncio/plugin.py", line 293, in _asyncgen_fixture_wrapper
result = event_loop.run_until_complete(setup())
File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/pytest_asyncio/plugin.py", line 275, in setup
res = await gen_obj.__anext__()
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/pytest_operator/plugin.py", line 231, in ops_test
await ops_test._setup_model()
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/pytest_operator/plugin.py", line 690, in _setup_model
model_state = await self._add_model(
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/pytest_operator/plugin.py", line 627, in _add_model
model = await controller.add_model(model_name, cloud_name, **kwargs)
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/juju/controller.py", line 342, in add_model
model_info = await model_facade.CreateModel(
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/juju/client/facade.py", line 481, in wrapper
reply = await f(*args, **kwargs)
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/juju/client/old_clients/_client9.py", line 3158, in CreateModel
reply = await self.rpc(msg)
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/juju/client/facade.py", line 654, in rpc
result = await self.connection.rpc(msg, encoder=TypeEncoder)
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/juju/client/connection.py", line 634, in rpc
raise errors.JujuAPIError(result)
juju.errors.JujuAPIError: failed to open kubernetes client: unable to determine legacy status for namespace test-charm-cmhu: Unauthorized
=================================================================================================== short test summary info ===================================================================================================
ERROR tests/integration/test_charm.py::test_active - juju.errors.JujuAPIError: failed to open kubernetes client: unable to determine legacy status for namespace test-charm-cmhu: Unauthorized
ERROR tests/integration/test_charm.py::test_build_succeeds - juju.errors.JujuAPIError: failed to open kubernetes client: unable to determine legacy status for namespace test-charm-cmhu: Unauthorized
====================================================================================================== 2 errors in 1.01s ======================================================================================================
Task was destroyed but it is pending!
task: <Task pending name='Task-21' coro=<WebSocketCommonProtocol.recv() done, defined at /home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py:486> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[create_task_with_handler.<locals>._task_result_exp_handler(task_name='tmp', logger=<Logger juju....ion (WARNING)>)() at /home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/juju/jasyncio.py:64]>
Task was destroyed but it is pending!
task: <Task pending name='Task-22' coro=<Event.wait() done, defined at /usr/lib/python3.10/asyncio/locks.py:201> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Task was destroyed but it is pending!
task: <Task pending name='Task-4' coro=<WebSocketCommonProtocol.transfer_data() running at /home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py:945> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[Task.task_wakeup()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-5' coro=<WebSocketCommonProtocol.keepalive_ping() running at /home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py:1234> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Task was destroyed but it is pending!
task: <Task pending name='Task-6' coro=<WebSocketCommonProtocol.close_connection() running at /home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py:1289> wait_for=<Task pending name='Task-4' coro=<WebSocketCommonProtocol.transfer_data() running at /home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py:945> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[Task.task_wakeup()]>>
Task was destroyed but it is pending!
task: <Task pending name='Task-7' coro=<Connection._receiver() running at /home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/juju/client/connection.py:531> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Task was destroyed but it is pending!
task: <Task pending name='Task-12' coro=<Connection._pinger() running at /home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/juju/client/connection.py:579> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Exception ignored in: <coroutine object WebSocketCommonProtocol.close_connection at 0x7fe9f6ef7450>
Traceback (most recent call last):
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 1329, in close_connection
await self.close_transport()
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 1347, in close_transport
if await self.wait_for_connection_lost():
File "/home/jdkandersson/src/jenkins-agent-operator/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 1372, in wait_for_connection_lost
await asyncio.wait_for(
File "/usr/lib/python3.10/asyncio/tasks.py", line 405, in wait_for
loop = events.get_running_loop()
RuntimeError: no running event loop
ERROR: InvocationError for command /home/jdkandersson/src/jenkins-agent-operator/.tox/integration/bin/pytest -v --tb native --ignore=/home/jdkandersson/src/jenkins-agent-operator/tests/unit --log-cli-level=INFO -s --jenkins-agent-image localhost:32000/jenkins-agent-k8s:85f19a452481f845507e9457796022a751aa039b --cloud micro-lxd --controller lxd --jenkins-controller-name lxd --jenkins-model-name jenkins --jenkins-unit-number 1 (exited with code 1)
___________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________
ERROR: integration: commands failed
I was able to deploy charms to the kubernetes controlled from lxd fine using normal juju commands.
What I got working was to have the integration tests run on a separate microk8s controller and use cross-model and cross-controller relations. However, this required the use of ops_test.juju because I could not find a way to create the relations natively e.g. using ops_test.model.relate if the relation is on another model (and in this case the model is on another controller).
Thanks for creating and maintaining the pytest operator, it makes integration tests much easier! This is a fairly broad feature request (or potentially a bug report). Recently I have been testing a charm on kubernetes that depends on another charm that can only run on lxd. The first thing I tried was to have a controller running on lxd and use add-k8s to also get the lxd controller to manage the kubernetes charm. Then I used the
--cloudand--controllerparameters to request pytest operator to use the lxd controller and kubernetes cloud. This resulted in the following error (I also tried this on GitHub actions with the same result, it looks like some weird unauthorized error):I was able to deploy charms to the kubernetes controlled from lxd fine using normal juju commands.
What I got working was to have the integration tests run on a separate microk8s controller and use cross-model and cross-controller relations. However, this required the use of
ops_test.jujubecause I could not find a way to create the relations natively e.g. usingops_test.model.relateif the relation is on another model (and in this case the model is on another controller).