search

stackabletech / issues

This repository is only for issues that concern multiple repositories or don't fit into any specific repository
2 stars 0 forks source link

Restructure operator Ci pipelines to improve build times #348

Open razvan opened 1 year ago

razvan commented 1 year ago

TODO

soenkeliebau commented 1 year ago

There already is https://github.com/stackabletech/operator-templating/pull/212 for this - which should pretty much work with a few minor updates and rebasing on main.

razvan commented 1 year ago

In the "cargo chef vs. crate2nix" meeting we agreed that the Docker based pipeline will remain the source of prod artifacts and that the Nix based pipeline will be used primarily for development until all implications on OpenShift compatibility and customer acceptance are clarified.

soenkeliebau commented 1 year ago

That being said, there are still things that we can do to improve build times. Pretty sure some actions still not use the shared cache for building, and we can pass the built image between actions to avoid constant rebuilding I think.

I seem to remember a ticket for this, but couldn't find it ....

fhennig commented 1 year ago

I've noticed that cargo udeps takes a long time. Could we maybe split build and publish, and already run the build in parallel to the tests? But just don't publish if the tests fail. We'd waste a build in case of failed tests, but if the tests pass, the whole thing would be much faster.

This thread also links some ways we could parallelize more, but still cancel workflows if any of the tests fail (i.e. fail fast across jobs)

fhennig commented 1 year ago

We should also try to just skip the docker build step as a whole if only tests or docs or other non-code files were changed.

fhennig commented 1 year ago

I found this snippet on how to build a Docker image in one job and share it with other jobs

fhennig commented 1 month ago

@razvan did we make some progress on this last week? I know a bunch of things changed in the CI

razvan commented 1 month ago

@razvan did we make some progress on this last week? I know a bunch of things changed in the CI

Unfortunately not much was done for the operator images. Most of the improvements were regarding the product image builds.