search

Qiskit / qiskit-serverless

A programming model for leveraging quantum and classical resources
https://qiskit.github.io/qiskit-serverless/
Apache License 2.0
61 stars 27 forks source link

Test building all images on PRs #1050

Closed psschwei closed 8 months ago

psschwei commented 9 months ago

We've run into issues (e.g. #1037) where some Python versions that we build images for end up failing when we cut a new release. Ideally, we should catch these images on the PR with a failing test, but because we don't test each Python version on a PR, we have the situation where our tests are passing when they probably shouldn't be :smile:

So, let's try building each version on each PR (we don't need to push the images though).

We could also expand the notebook test to run for multiple versions, so that we're not just building but also running those images. Ideally we'd do both...

xref the discussion in #1046

Tansito commented 9 months ago

Something that I did in the past was to separate PRs for releases and normal PRs. Thinking that some actions could take more time than we would like (like the vscan for our docker images). So what I do is to check that but in the release pull request:

https://github.com/Qiskit-Extensions/quantum-serverless/blob/main/.github/workflows/release-pull-request.yaml

Maybe we can think in something like this before a release if we see that it could be problematic to have so much actions in every pull request for every change.

akihikokuroda commented 8 months ago

How about running some actions (like simulate the release process) every evening so that it can catch breaks?

psschwei commented 8 months ago

It'd be nice to run on every PR so as to catch breaks before they happen (rather than try to triage which commit caused it after the fact, especially if there's a bunch of commits in the next release).

Looking at past runs, the builds take under an hour (and that includes pushing to ICR which takes at least a few minutes)... that doesn't seem too bad to me. We could also probably tweak that job to run in parallel and drop down to 10-15 minutes...