search

canonical / charm-relation-interfaces

https://canonical.github.io/charm-relation-interfaces/
Apache License 2.0
16 stars 48 forks source link

feat: charm relation interfaces tests running on a schedule #153

Closed IronCore864 closed 2 months ago

IronCore864 commented 4 months ago

Run charm relation interface tests on a schedule.

Changes

Investigation

Interfaces with interface_tests

Charms with Conftests

Interfaces Use Charms with Conftests

Interfaces Both Have interface_tests and Use Charms with Conftests

So, as a summary, currently, only tests for the following interfaces could run theoretically:

Manual Tests Result

TODO

benhoyt commented 4 months ago

Thanks!

Where will the errors/output of this end up? Will we or someone be notified when there are failures?

Could you run this on your branch and show us an example of the output? I see the last run was over a year ago, and GitHub has pruned the logs.

IronCore864 commented 4 months ago

Sample output for one interface, pass:

$ python3 run_matrix.py --include ingress
Failed to load module test_requirer: No module named 'test_requirer'
Failed to load module test_requirer: No module named 'test_requirer'
INFO:root:Running tests for interface: ingress
INFO:root:Running tests for version: v1
INFO:root:Running tests for role: provider
INFO:root:Running 4 ingress interface tests on: ['traefik-k8s']...
INFO:root:Running tests for ingress
INFO:root:Running tests for charm: traefik-k8s
INFO:root:Preparing testing environment for: traefik-k8s
INFO:root:Cloning: traefik-k8s from (https://github.com/canonical/traefik-k8s-operator@main)
INFO:root:Preparing venv for /tmp/charm-relation-interfaces-tests/traefik-k8s
INFO:root:Installing dependencies in venv for /tmp/charm-relation-interfaces-tests/traefik-k8s
INFO:root:Installed # Cryptographic primitives and recipes
# Code: https://github.com/pyca/cryptography
# Docs: https://cryptography.io/
# Deps: tls-certificates-interface
cryptography

# Handle merging of nested data structures in python.
# Code: https://github.com/toumorokoshi/deepmerge
# Docs: https://deepmerge.readthedocs.io/en/latest/
# Deps: charm
deepmerge

# An implementation of the JSON Schema specification
# Code: https://github.com/python-jsonschema/jsonschema
# Docs: https://python-jsonschema.readthedocs.io/
# Deps: traefik_k8s libs, tls-certificates-interface
jsonschema

# Lightweight k8s module
# Code: https://github.com/gtsystem/lightkube
# Docs: https://lightkube.readthedocs.io/
# Deps: charm, observability_libs
lightkube >= 0.8.1
lightkube-models >= 1.22.0.4

# Operator Framework
# Code: https://github.com/canonical/operator/
# Docs: https://ops.rtfd.io/
# Deps: charm
ops >= 2.10.0

# the deps below are from PYDEPS and are needed here for the tox test envs
# Keep them in sync with pydeps!
importlib-metadata==6.0.0
opentelemetry-exporter-otlp-proto-http==1.21.0
pydantic>=2

cosl

INFO:root:Generating test file for ingress at /tmp/charm-relation-interfaces-tests/traefik-k8s/tests/interface
INFO:root:Running tests for /tmp/charm-relation-interfaces-tests/traefik-k8s
===================================================================== test session starts ======================================================================
platform darwin -- Python 3.11.8, pytest-8.2.2, pluggy-1.5.0
rootdir: /tmp/charm-relation-interfaces-tests/traefik-k8s
configfile: pyproject.toml
plugins: anyio-4.4.0, interface-tester-2.0.1
collected 1 item

../../../../tmp/charm-relation-interfaces-tests/traefik-k8s/tests/interface/interface_test_ingress.py .                                                  [100%]

======================================================================= warnings summary =======================================================================
lib/charms/traefik_k8s/v2/ingress.py:255
  /private/tmp/charm-relation-interfaces-tests/traefik-k8s/lib/charms/traefik_k8s/v2/ingress.py:255: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
    @validator("scheme", pre=True)

lib/charms/traefik_k8s/v2/ingress.py:262
  /private/tmp/charm-relation-interfaces-tests/traefik-k8s/lib/charms/traefik_k8s/v2/ingress.py:262: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
    @validator("port", pre=True)

lib/charms/traefik_k8s/v2/ingress.py:280
  /private/tmp/charm-relation-interfaces-tests/traefik-k8s/lib/charms/traefik_k8s/v2/ingress.py:280: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
    @validator("host", pre=True)

lib/charms/traefik_k8s/v2/ingress.py:286
  /private/tmp/charm-relation-interfaces-tests/traefik-k8s/lib/charms/traefik_k8s/v2/ingress.py:286: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
    @validator("ip", pre=True)

.interface-venv/lib/python3.11/site-packages/_pytest/config/__init__.py:1448
  /private/tmp/charm-relation-interfaces-tests/traefik-k8s/.interface-venv/lib/python3.11/site-packages/_pytest/config/__init__.py:1448: PytestConfigWarning: Unknown config option: asyncio_mode

    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

tests/interface/interface_test_ingress.py::test_ingress_interface
  /var/folders/2j/5hwf_xns24z3hc6bmf268x340000gn/T/tmpkvddgiuz/charm-relation-interfaces/interfaces/ingress/v1/schema.py:37: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/

tests/interface/interface_test_ingress.py: 31 warnings
  /private/tmp/charm-relation-interfaces-tests/traefik-k8s/.interface-venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py:273: PydanticDeprecatedSince20: The `__fields__` attribute is deprecated, use `model_fields` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================ 1 passed, 37 warnings in 1.99s ================================================================
INFO:root:Result: PASSED
INFO:root:Running tests for role: requirer
INFO:root:No tests specified for ingress/requirer; skipping...
INFO:root:Running tests for version: v2
INFO:root:Running tests for role: provider
INFO:root:Running 5 ingress interface tests on: ['traefik-k8s']...
INFO:root:Running tests for ingress
INFO:root:Running tests for charm: traefik-k8s
INFO:root:Preparing testing environment for: traefik-k8s
INFO:root:Generating test file for ingress at /tmp/charm-relation-interfaces-tests/traefik-k8s/tests/interface
INFO:root:Running tests for /tmp/charm-relation-interfaces-tests/traefik-k8s
===================================================================== test session starts ======================================================================
platform darwin -- Python 3.11.8, pytest-8.2.2, pluggy-1.5.0
rootdir: /tmp/charm-relation-interfaces-tests/traefik-k8s
configfile: pyproject.toml
plugins: anyio-4.4.0, interface-tester-2.0.1
collected 1 item

../../../../tmp/charm-relation-interfaces-tests/traefik-k8s/tests/interface/interface_test_ingress.py .                                                  [100%]

======================================================================= warnings summary =======================================================================
lib/charms/traefik_k8s/v2/ingress.py:255
  /private/tmp/charm-relation-interfaces-tests/traefik-k8s/lib/charms/traefik_k8s/v2/ingress.py:255: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
    @validator("scheme", pre=True)

lib/charms/traefik_k8s/v2/ingress.py:262
  /private/tmp/charm-relation-interfaces-tests/traefik-k8s/lib/charms/traefik_k8s/v2/ingress.py:262: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
    @validator("port", pre=True)

lib/charms/traefik_k8s/v2/ingress.py:280
  /private/tmp/charm-relation-interfaces-tests/traefik-k8s/lib/charms/traefik_k8s/v2/ingress.py:280: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
    @validator("host", pre=True)

lib/charms/traefik_k8s/v2/ingress.py:286
  /private/tmp/charm-relation-interfaces-tests/traefik-k8s/lib/charms/traefik_k8s/v2/ingress.py:286: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
    @validator("ip", pre=True)

.interface-venv/lib/python3.11/site-packages/_pytest/config/__init__.py:1448
  /private/tmp/charm-relation-interfaces-tests/traefik-k8s/.interface-venv/lib/python3.11/site-packages/_pytest/config/__init__.py:1448: PytestConfigWarning: Unknown config option: asyncio_mode

    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

tests/interface/interface_test_ingress.py: 49 warnings
  /private/tmp/charm-relation-interfaces-tests/traefik-k8s/.interface-venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py:273: PydanticDeprecatedSince20: The `__fields__` attribute is deprecated, use `model_fields` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
    warnings.warn(

tests/interface/interface_test_ingress.py::test_ingress_interface
tests/interface/interface_test_ingress.py::test_ingress_interface
tests/interface/interface_test_ingress.py::test_ingress_interface
tests/interface/interface_test_ingress.py::test_ingress_interface
  /private/tmp/charm-relation-interfaces-tests/traefik-k8s/.interface-venv/lib/python3.11/site-packages/pydantic/main.py:1070: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
    warnings.warn('The `dict` method is deprecated; use `model_dump` instead.', category=PydanticDeprecatedSince20)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================ 1 passed, 58 warnings in 1.72s ================================================================
INFO:root:Result: PASSED
INFO:root:Running tests for role: requirer
INFO:root:No tests specified for ingress/requirer; skipping...
+++ Results +++
{
  "ingress": {
    "v1": {
      "provider": {
        "traefik-k8s": true
      },
      "requirer": {}
    },
    "v2": {
      "provider": {
        "traefik-k8s": true
      },
      "requirer": {}
    }
  }
}

Sample output for one interface, fail:

$ python3 run_matrix.py --include smtp
Failed to load module test_requirer: No module named 'test_requirer'
INFO:root:Running tests for interface: smtp
INFO:root:Running tests for version: v0
INFO:root:Running tests for role: provider
INFO:root:Running 1 smtp interface tests on: ['smtp-integrator']...
INFO:root:Running tests for smtp
INFO:root:Running tests for charm: smtp-integrator
INFO:root:Preparing testing environment for: smtp-integrator
INFO:root:Cloning: smtp-integrator from (https://github.com/canonical/smtp-integrator-operator@main)
INFO:root:Preparing venv for /tmp/charm-relation-interfaces-tests/smtp-integrator
INFO:root:Installing dependencies in venv for /tmp/charm-relation-interfaces-tests/smtp-integrator
INFO:root:Installed ops==2.14.0
pydantic==2.7.4

INFO:root:Generating test file for smtp at /tmp/charm-relation-interfaces-tests/smtp-integrator/tests/interface
INFO:root:Running tests for /tmp/charm-relation-interfaces-tests/smtp-integrator
===================================================================== test session starts ======================================================================
platform darwin -- Python 3.11.8, pytest-8.2.2, pluggy-1.5.0
rootdir: /tmp/charm-relation-interfaces-tests/smtp-integrator
configfile: pyproject.toml
plugins: interface-tester-2.0.1
collected 1 item

../../../../tmp/charm-relation-interfaces-tests/smtp-integrator/tests/interface/interface_test_smtp.py F                                                 [100%]

=========================================================================== FAILURES ===========================================================================
_____________________________________________________________________ test_smtp_interface ______________________________________________________________________

interface_tester = <Interface Tester:
            repo=https://github.com/IronCore864/charm-relation-interfaces
            branch=my-fancy-da...d_units=1, deferred=[], stored_state=[], app_status=UnknownStatus(), unit_status=UnknownStatus(), workload_version='')>

    def test_smtp_interface(interface_tester: InterfaceTester):
        interface_tester.configure(
            interface_name="smtp",
            interface_version=0,
            repo="https://github.com/IronCore864/charm-relation-interfaces",
            branch="my-fancy-database",
        )
>       interface_tester.run()

/tmp/charm-relation-interfaces-tests/smtp-integrator/tests/interface/interface_test_smtp.py:11:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Interface Tester:
            repo=https://github.com/IronCore864/charm-relation-interfaces
            branch=my-fancy-da...d_units=1, deferred=[], stored_state=[], app_status=UnknownStatus(), unit_status=UnknownStatus(), workload_version='')>

    def run(self) -> bool:
        """Run interface tests.

        Returns True if some tests were found and ran, False otherwise.
        """
        self._validate_config()  # will raise if misconfigured
        logger.info(f"Running {repr(self)}.")
        errors = []
        ran_some = False

        for test_fn, role, schema in self._yield_tests():
            ctx = _InterfaceTestContext(
                role=role,
                schema=schema,
                interface_name=self._interface_name,
                version=self._interface_version,
                charm_type=self._charm_type,
                state_template=self._state_template,
                meta=self.meta,
                config=self.config,
                actions=self.actions,
                supported_endpoints=self._gather_supported_endpoints(),
                test_fn=test_fn,
                juju_version=self._juju_version,
            )
            try:
                with tester_context(ctx):
                    test_fn()
            except Exception as e:
                logger.exception(f"Interface tester plugin failed with {e}")

                if self._RAISE_IMMEDIATELY:
                    raise e

                errors.append((ctx, e))
            ran_some = True

        # todo: consider raising custom exceptions here.
        if errors:
            msgs = []
            for ctx, e in errors:
                msgs.append(
                    f" - {ctx.interface_name}[v{ctx.version}]@{ctx.role}:{ctx.test_fn} raised {e}"
                )
            long_msg = "\n".join(msgs)

>           raise InterfaceTestsFailed(
                f"interface tests completed with {len(errors)} errors. \n" + long_msg
            )
E           interface_tester.errors.InterfaceTestsFailed: interface tests completed with 1 errors.
E            - smtp[v0]@provider:<function test_data_published_on_created at 0x1045f7880> raised Multiple endpoints found for provider/smtp: ['smtp', 'smtp-legacy']: cannot guess which one it is we're supposed to be testing

.interface-venv/lib/python3.11/site-packages/interface_tester/plugin.py:347: InterfaceTestsFailed
---------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------
INFO     pytest_interface_tester:plugin.py:307 Running <Interface Tester:
            repo=https://github.com/IronCore864/charm-relation-interfaces
            branch=my-fancy-database
            base_path=interfaces
            charm_type=<class 'charm.SmtpIntegratorOperatorCharm'>
            meta=None
            actions=None
            config=None
            interface_name=smtp
            interface_version=0
            juju_version=None
            state_template=State(config={'host': 'smtp.example'}, relations=[PeerRelation(endpoint='smtp-peers', interface=None, relation_id=1, local_app_data={}, local_unit_data={'egress-subnets': ' 192.0.2.0', 'ingress-address': ' 192.0.2.0', 'private-address': ' 192.0.2.0'}, peers_data={0: {'egress-subnets': ' 192.0.2.0', 'ingress-address': ' 192.0.2.0', 'private-address': ' 192.0.2.0'}})], networks={}, containers=[], storage=[], opened_ports=[], leader=True, model=Model(name='2r467aMGw3FTWxCaID2u', uuid='cb4ae41e-df7b-4c89-a18c-f16c30fc3ab8', type='kubernetes', cloud_spec=None), secrets=[], resources={}, planned_units=1, deferred=[], stored_state=[], app_status=UnknownStatus(), unit_status=UnknownStatus(), workload_version='')>.
WARNING  interface_tests_checker:collector.py:227 Failed to load module test_requirer: No module named 'test_requirer'
WARNING  pytest_interface_tester:plugin.py:245 skipping role requirer: unsupported by this charm.
WARNING  pytest_interface_tester:plugin.py:245 skipping role requirer: unsupported by this charm.
ERROR    pytest_interface_tester:plugin.py:330 Interface tester plugin failed with Multiple endpoints found for provider/smtp: ['smtp', 'smtp-legacy']: cannot guess which one it is we're supposed to be testing
Traceback (most recent call last):
  File "/private/tmp/charm-relation-interfaces-tests/smtp-integrator/.interface-venv/lib/python3.11/site-packages/interface_tester/plugin.py", line 328, in run
    test_fn()
  File "/var/folders/2j/5hwf_xns24z3hc6bmf268x340000gn/T/tmpn2zbozv9/charm-relation-interfaces/interfaces/smtp/v0/interface_tests/test_provider.py", line 18, in test_data_published_on_created
  File "/private/tmp/charm-relation-interfaces-tests/smtp-integrator/.interface-venv/lib/python3.11/site-packages/interface_tester/interface_test.py", line 266, in run
    state_out = self._run(event)
                ^^^^^^^^^^^^^^^^
  File "/private/tmp/charm-relation-interfaces-tests/smtp-integrator/.interface-venv/lib/python3.11/site-packages/interface_tester/interface_test.py", line 384, in _run
    relations = self._generate_relations_state(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/charm-relation-interfaces-tests/smtp-integrator/.interface-venv/lib/python3.11/site-packages/interface_tester/interface_test.py", line 469, in _generate_relations_state
    endpoint = self._get_endpoint(supported_endpoints, role, interface_name=interface_name)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/charm-relation-interfaces-tests/smtp-integrator/.interface-venv/lib/python3.11/site-packages/interface_tester/interface_test.py", line 450, in _get_endpoint
    raise ValueError(
ValueError: Multiple endpoints found for provider/smtp: ['smtp', 'smtp-legacy']: cannot guess which one it is we're supposed to be testing
=================================================================== short test summary info ====================================================================
FAILED ../../../../tmp/charm-relation-interfaces-tests/smtp-integrator/tests/interface/interface_test_smtp.py::test_smtp_interface - interface_tester.errors.InterfaceTestsFailed: interface tests completed with 1 errors.
====================================================================== 1 failed in 1.52s =======================================================================
WARNING:root:interface tests for smtp-integrator smtp provider failed
Traceback (most recent call last):
  File "/Users/tiexin/work/charm-relation-interfaces/run_matrix.py", line 197, in _run_test_with_pytest
    subprocess.check_call(
  File "/Users/tiexin/.pyenv/versions/3.11.8/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'PYTHONPATH=src:lib .interface-venv/bin/python -m pytest /tmp/charm-relation-interfaces-tests/smtp-integrator/tests/interface/interface_test_smtp.py' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/tiexin/work/charm-relation-interfaces/run_matrix.py", line 228, in _test_charm
    _run_test_with_pytest(charm_path, test_path)
  File "/Users/tiexin/work/charm-relation-interfaces/run_matrix.py", line 202, in _run_test_with_pytest
    raise InterfaceTestError from e
InterfaceTestError
INFO:root:Result: FAILED
INFO:root:Running tests for role: requirer
INFO:root:No tests specified for smtp/requirer; skipping...
+++ Results +++
{
  "smtp": {
    "v0": {
      "provider": {
        "smtp-integrator": false
      },
      "requirer": {}
    }
  }
}

I have not run the whole suite yet because most fail, and the output is a very long output like the above but repeated multiple times.

tonyandrewmeyer commented 4 months ago

This was done for better charmhub support. I couldn't easily see why the rename was important for that, but I assume it is for some reason. So probably we actually need to update pytest-interface-tester instead and not rename (there's a comment along those lines on that PR too).

IronCore864 commented 3 months ago

After discussion, we decide to create issues for failed interface tests and assign them to owners. (Previously we wanted to use email notification; after discussion, we decided GitHub issues fit better.)

To create issues and assign them to owners, a map between interfaces to owners is required. This PR adds support to it, and see the discussion here for the rationale.

The changes in the current PR are tested in a separate repo. See the succeeded GitHub Actions workflow run here. This issue is created automatically by the GitHub Actions workflow and assigned to the owner of the interface. Issues won't be created for failed interfaces without owners in their interface.yaml definition.

IronCore864 commented 3 months ago

All comments were resolved, and I also added the feature to use labels when searching for and creating issues.

IronCore864 commented 3 months ago

Waiting for https://github.com/canonical/pytest-interface-tester/pull/19 to be merged first so that we can use pytest-interface-tester v3.1.0 directly instead of using a fork.

IronCore864 commented 2 months ago

Updated lib version to pytest-interface-tester>=3.1.0 in pyproject.toml, test passed: https://github.com/IronCore864/charm-relation-interfaces/actions/runs/10348534960/job/28640972468, issues can be created and labeled: https://github.com/IronCore864/charm-relation-interfaces/issues/6.

Please go ahead and merge @benhoyt @PietroPasotti