search

canonical / kfp-operators

Kubeflow Pipelines Operators
Apache License 2.0
2 stars 12 forks source link

Move charm building from pytest tests to separate workflow jobs to increase CI stability #448

Open ca-scribner opened 5 months ago

ca-scribner commented 5 months ago

Context

We've encountered a lot of instability in the kfp CI of late, including oddly flaky charmcraft builds for some of the charms. For some unknown reasons, standalone charmcraft seems to build things more reliably than building them in the pytest file directly using the pytest-operator.

Proposed here is that we move the building for these charms out of the pytest files and into their own jobs, then pass the .charm files as artifacts to the tests/publish jobs. This would hopefully add stability just from using charmcraft, but also we can have retry logic around failed build jobs (build jobs could auto-retry X times, so flakes could be mitigated)

What needs to get done

  1. refactor the pytest tests to optionally build the charms or accept their artifacts as inputs. We might be able to use some data platform CI as inspiration here
  2. refactor the CI workflows to build the charms separately and pass them to integration tests

Definition of Done

  1. kfp integration test CI does not build its own charms, but instead uses those built in separate jobs
syncronize-issues-to-jira[bot] commented 5 months ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5618.

This message was autogenerated

ca-scribner commented 5 months ago

canonical/bundle-kubeflow#1005 discusses instability that feels related to this issue