search

canonical / kfp-operators

Kubeflow Pipelines Operators
Apache License 2.0
2 stars 12 forks source link

Integration tests for `kfp-api` not working for `track/2.0-alpha.7` #418

Open misohu opened 5 months ago

misohu commented 5 months ago

Bug Description

Integration test for kfp-api are not working for track/2.0-alpha.7. After executing the tox environment for integration tests they fail while trying to deploy built kfp-api charm to juju with message.

====================================== short test summary info =======================================
FAILED tests/integration/test_charm.py::TestCharm::test_build_and_deploy - juju.errors.JujuError: ['charm "local:focal/kfp-api-7" not found']

You can narrow the problem down just by running the test_build_and_deploy test case.

To Reproduce

  1. clone repo
  2. git checkout track/2.0-alpha.7
  3. cd charms/kfp-api
  4. tox -e integration -- --model kubeflow --keep-models -vv -s

Environment

juju - 2.9.47-ubuntu-amd64 microk8s - v1.24.17

Relevant Log Output

ubuntu@ip-172-31-22-232:~/kfp-operators/charms/kfp-api$ tox -e integration -- --model kubeflow --keep-models -vv -s
integration: install_deps> python -I -m pip install -r requirements-integration.txt
integration: commands[0]> pytest -vv --tb native --asyncio-mode=auto /home/ubuntu/kfp-operators/charms/kfp-api/tests/integration --log-cli-level=INFO -s --model kubeflow --keep-models -vv -s
======================================== test session starts =========================================
platform linux -- Python 3.8.10, pytest-7.4.2, pluggy-1.3.0 -- /home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/bin/python
cachedir: .tox/integration/.pytest_cache
rootdir: /home/ubuntu/kfp-operators/charms/kfp-api
configfile: pyproject.toml
plugins: operator-0.29.0, asyncio-0.21.1, anyio-4.0.0
asyncio: mode=auto
collected 6 items                                                                                    

tests/integration/test_charm.py::TestCharm::test_build_and_deploy 
------------------------------------------- live log setup -------------------------------------------
INFO     pytest_operator.plugin:plugin.py:675 Connecting to existing model github-pr-21c0c-microk8s:kubeflow on unspecified cloud
------------------------------------------- live log call --------------------------------------------
INFO     pytest_operator.plugin:plugin.py:526 Using tmp_path: /home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/tmp/pytest/kubeflow0
INFO     pytest_operator.plugin:plugin.py:975 Building charm kfp-api
INFO     pytest_operator.plugin:plugin.py:980 Built charm kfp-api in 40.94s
INFO     test_charm:test_charm.py:30 Built charm /home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/tmp/pytest/kubeflow0/charms/kfp-api_ubuntu-20.04-amd64.charm
INFO     juju.model:model.py:1993 Deploying local:kfp-api-7
FAILED
tests/integration/test_charm.py::TestCharm::test_relational_db_relation_with_mysql_relation XFAIL
tests/integration/test_charm.py::TestCharm::test_relational_db_relation_with_mysql_k8s XFAIL
tests/integration/test_charm.py::TestCharm::test_msql_relation_with_relational_db_relation XFAIL
tests/integration/test_charm.py::TestCharm::test_prometheus_grafana_integration XFAIL
tests/integration/test_charm.py::TestCharm::test_remove_application XFAIL (aborted)
----------------------------------------- live log teardown ------------------------------------------
INFO     pytest_operator.plugin:plugin.py:783 Model status:

Model     Controller                Cloud/Region        Version  SLA          Timestamp
kubeflow  github-pr-21c0c-microk8s  microk8s/localhost  2.9.47   unsupported  11:47:47Z

INFO     pytest_operator.plugin:plugin.py:789 Juju error logs:

unit-kfp-api-0: 12:38:55 ERROR unit.kfp-api/0.juju-log Failed to generate container configuration.
unit-kfp-api-0: 12:38:55 ERROR unit.kfp-api/0.juju-log Failed to handle <LeaderElectedEvent via KfpApiOperator/on/leader_elected[6]> with error: Please add required database relation: eg. relational-db
unit-kfp-api-0: 12:38:56 ERROR unit.kfp-api/0.juju-log Failed to generate container configuration.
unit-kfp-api-0: 12:38:56 ERROR unit.kfp-api/0.juju-log Failed to handle <PebbleReadyEvent via KfpApiOperator/on/apiserver_pebble_ready[11]> with error: Please add required database relation: eg. relational-db
unit-kfp-api-0: 12:38:57 ERROR unit.kfp-api/0.juju-log Failed to generate container configuration.
unit-kfp-api-0: 12:38:57 ERROR unit.kfp-api/0.juju-log Failed to handle <ConfigChangedEvent via KfpApiOperator/on/config_changed[16]> with error: Please add required database relation: eg. relational-db
unit-kfp-api-0: 12:43:16 ERROR unit.kfp-api/0.juju-log Failed to generate container configuration.
unit-kfp-api-0: 12:43:16 ERROR unit.kfp-api/0.juju-log Failed to handle <UpdateStatusEvent via KfpApiOperator/on/update_status[26]> with error: Please add required database relation: eg. relational-db
unit-kfp-api-0: 09:06:16 ERROR unit.kfp-api/0.juju-log Failed to generate container configuration.
unit-kfp-api-0: 09:06:16 ERROR unit.kfp-api/0.juju-log Failed to handle <LeaderElectedEvent via KfpApiOperator/on/leader_elected[6]> with error: Please add required database relation: eg. relational-db
unit-kfp-api-0: 09:06:17 ERROR unit.kfp-api/0.juju-log Failed to generate container configuration.
unit-kfp-api-0: 09:06:17 ERROR unit.kfp-api/0.juju-log Failed to handle <PebbleReadyEvent via KfpApiOperator/on/apiserver_pebble_ready[11]> with error: Please add required database relation: eg. relational-db
unit-kfp-api-0: 09:06:18 ERROR unit.kfp-api/0.juju-log Failed to generate container configuration.
unit-kfp-api-0: 09:06:18 ERROR unit.kfp-api/0.juju-log Failed to handle <ConfigChangedEvent via KfpApiOperator/on/config_changed[16]> with error: Please add required database relation: eg. relational-db

INFO     pytest_operator.plugin:plugin.py:855 Forgetting main...

============================================== FAILURES ==============================================
__________________________________ TestCharm.test_build_and_deploy ___________________________________
Traceback (most recent call last):
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 177, in pytest_runtest_call
    raise e
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
    item.runtest()
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 1792, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 532, in inner
    _loop.run_until_complete(task)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/ubuntu/kfp-operators/charms/kfp-api/tests/integration/test_charm.py", line 35, in test_build_and_deploy
    await ops_test.model.deploy(
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/juju/model.py", line 1750, in deploy
    return await self._deploy(
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/juju/model.py", line 2019, in _deploy
    result = await app_facade.Deploy(applications=[app])
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 486, in wrapper
    reply = await f(*args, **kwargs)
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/juju/client/_client13.py", line 1107, in Deploy
    reply = await self.rpc(msg)
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 659, in rpc
    result = await self.connection.rpc(msg, encoder=TypeEncoder)
  File "/home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 652, in rpc
    raise errors.JujuError(err_results)
juju.errors.JujuError: ['charm "local:focal/kfp-api-7" not found']
----------------------------------------- Captured log setup -----------------------------------------
INFO     pytest_operator.plugin:plugin.py:675 Connecting to existing model github-pr-21c0c-microk8s:kubeflow on unspecified cloud
----------------------------------------- Captured log call ------------------------------------------
INFO     pytest_operator.plugin:plugin.py:526 Using tmp_path: /home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/tmp/pytest/kubeflow0
INFO     pytest_operator.plugin:plugin.py:975 Building charm kfp-api
INFO     pytest_operator.plugin:plugin.py:980 Built charm kfp-api in 40.94s
INFO     test_charm:test_charm.py:30 Built charm /home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/tmp/pytest/kubeflow0/charms/kfp-api_ubuntu-20.04-amd64.charm
INFO     juju.model:model.py:1993 Deploying local:kfp-api-7
========================================== warnings summary ==========================================
tests/integration/test_charm.py::TestCharm::test_build_and_deploy
tests/integration/test_charm.py::TestCharm::test_build_and_deploy
  /home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:218: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    self._drain_lock = asyncio.Lock(loop=loop)

tests/integration/test_charm.py::TestCharm::test_build_and_deploy
tests/integration/test_charm.py::TestCharm::test_build_and_deploy
tests/integration/test_charm.py::TestCharm::test_build_and_deploy
tests/integration/test_charm.py::TestCharm::test_build_and_deploy
tests/integration/test_charm.py::TestCharm::test_build_and_deploy
tests/integration/test_charm.py::TestCharm::test_build_and_deploy
  /home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:977: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.sleep(self.ping_interval, loop=self.loop)

tests/integration/test_charm.py: 35 warnings
  /home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:911: DeprecationWarning: 'with (yield from lock)' is deprecated use 'async with lock' instead
    with (yield from self._drain_lock):

tests/integration/test_charm.py: 30 warnings
  /home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:416: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait(

tests/integration/test_charm.py::TestCharm::test_build_and_deploy
tests/integration/test_charm.py::TestCharm::test_build_and_deploy
tests/integration/test_charm.py::TestCharm::test_build_and_deploy
tests/integration/test_charm.py::TestCharm::test_build_and_deploy
  /home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:988: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_charm.py::TestCharm::test_remove_application
tests/integration/test_charm.py::TestCharm::test_remove_application
  /home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:532: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_charm.py::TestCharm::test_remove_application
tests/integration/test_charm.py::TestCharm::test_remove_application
  /home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:554: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_charm.py::TestCharm::test_remove_application
tests/integration/test_charm.py::TestCharm::test_remove_application
  /home/ubuntu/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:1077: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
====================================== short test summary info =======================================
FAILED tests/integration/test_charm.py::TestCharm::test_build_and_deploy - juju.errors.JujuError: ['charm "local:focal/kfp-api-7" not found']
============================= 1 failed, 5 xfailed, 83 warnings in 56.10s =============================
integration: exit 1 (57.37 seconds) /home/ubuntu/kfp-operators/charms/kfp-api> pytest -vv --tb native --asyncio-mode=auto /home/ubuntu/kfp-operators/charms/kfp-api/tests/integration --log-cli-level=INFO -s --model kubeflow --keep-models -vv -s pid=2454997
  integration: FAIL code 1 (78.12=setup[20.75]+cmd[57.37] seconds)
  evaluation failed :( (78.21 seconds)

Additional Context

Things I tried:

syncronize-issues-to-jira[bot] commented 5 months ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5518.

This message was autogenerated

misohu commented 5 months ago

Juju team discussion here

misohu commented 5 months ago

After discussions with Juju team we managed to find fix by pinning the actions operator to 2.9/candidate. There is a launchpad bug tracking this issue here fix should be available in 2.9/stable soon.

misohu commented 5 months ago

After fixing the juju problem another problem emerged. The CI had pinned prometheus and grafana to latest/stable this cause incompatibility with juju 2.9. I managed to fifnd proper revisions for the charms. Solution was to check this related issue https://github.com/canonical/bundle-kubeflow/issues/688#issuecomment-1702810636

The error longs for the problem:

=================================== FAILURES ===================================
________________ TestCharm.test_prometheus_grafana_integration _________________
Traceback (most recent call last):
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 177, in pytest_runtest_call
    raise e
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
    item.runtest()
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 1792, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 532, in inner
    _loop.run_until_complete(task)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/runner/work/kfp-operators/kfp-operators/charms/kfp-api/tests/integration/test_charm.py", line 188, in test_prometheus_grafana_integration
    await ops_test.model.deploy(prometheus, channel="latest/stable", trust=True)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/juju/model.py", line 1737, in deploy
    return await self._deploy(
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/juju/model.py", line 2010, in _deploy
    result = await app_facade.Deploy(applications=[app])
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 486, in wrapper
    reply = await f(*args, **kwargs)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/juju/client/_client13.py", line 1107, in Deploy
    reply = await self.rpc(msg)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 659, in rpc
    result = await self.connection.rpc(msg, encoder=TypeEncoder)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 649, in rpc
    raise errors.JujuError(err_results)
juju.errors.JujuError: ['Charm feature requirements cannot be met:\n  - charm requires feature "juju" (version >= 3.0.3) but model currently supports version 2.9.48\n\nFeature descriptions:\n  - "juju": the version of Juju used by the model\n\nFor additional information please see: https://juju.is/docs/olm/supported-features']