search

shipwright-io / build

Shipwright - a framework for building container images on Kubernetes
https://shipwright.io
Apache License 2.0
669 stars 113 forks source link

parallel / serial execution policies in build v2 #56

Open gabemontero opened 4 years ago

gabemontero commented 4 years ago

Today build v1 allows for specification of these run policies at the build level

Tekton has

build v2 should surface some flavor / combinations of these in its API ... either leverage the tekton features as is, or add build v1 type controls in the build v2 controller.

sbose78 commented 4 years ago

Gabe, are you talking about providing a way to specify

gabemontero commented 4 years ago

Builds in parallel / serial is what buildv1 has today and what I'm referring to

Is it possible to provide the same with tekton as a base given what it provides? Or perhaps the question is "how expensive" would it be to do it given tekton as a base vs. "how desired" such a feature is ?

Given what tekton provides, feels like it would have to be achieved at the buildv2 operator level, where it monitors the state of each buildv2 build under a buildv2 buildconfig (assuming analogous API concepts in buildv2).

Right now I only see build and build strategy in https://github.com/redhat-developer/build/blob/master/pkg/apis/build/v1alpha1/build_types.go .. no "buildconfig"

So how would you group "builds" such that you decide if they run in parallel or serial?

Maybe the build specs get an "identifier" and you group different builds based on the fact that they use the same build spec?

stuff to sort out if it is decided buildv2 supports this .... otherwise this falls into the buildv2 vs buildv1 comparison doc page.

adambkaplan commented 4 years ago

The need for this is related to our trigger stories (#51, #52). Otherwise it is safer for us to assume users can control when builds get executed.

qu1queee commented 4 years ago

@gabemontero take for example the following scenario:

A user defines a Build X. Then the same user cooks two BuildRuns that reference Build X.

from the above, is up to the user to define when should those BuildRuns get trigger ( same as @adambkaplan mentioned). The user can opt for a serial approach or a parallel approach by some sort of means. E.g. in https://github.com/homeport/build-load buildruns are triggered in parallel.

gabemontero commented 4 years ago

yep @qu1queee .... and fwiw @adambkaplan 's https://github.com/shipwright-io/build/issues/56#issuecomment-718849155 stemmed from a team meeting we had to discuss this particular item and how best, if at all, shipwright provides something equivalent

most likely then when work on those trigger related stories evolve, they'll consider the parallel/serial element when a user is not directly responsible triggering buildruns, and when they do, we can close this issue in favor of where those are headed

gabemontero commented 3 years ago

Update: upstream tekton, as part of moving toward v1 of the API, have registered issues 3563 and 3052 (I am intentionally avoiding putting an actual link to avoid cross referencing for now to avoid clutter on the tekton side) that are in this space

qu1queee commented 3 years ago

Can we close this issue in favor of https://github.com/shipwright-io/build/issues/56#issuecomment-718849155 and https://github.com/shipwright-io/build/issues/56#issuecomment-727895129 ? Or do we want to keep this as a reminder of the ongoing Tekton efforts mentioned in https://github.com/shipwright-io/build/issues/56#issuecomment-735926704 ?

gabemontero commented 3 years ago

Yeah let's keep this as a reminder i.e. monitoring tekton. FWIW it is already in the POST GA milestone

gabemontero commented 3 years ago

So update on this one since the end of 2020 / beginning on 2021.

After following literally 6 or 7 opened and close links in upstream tekton, the current open / upstream tekton TEP in this space is TEP 44, decouple task composition from scheduling https://github.com/tektoncd/community/blob/main/teps/0044-decouple-task-composition-from-scheduling.md

However, a Task / Pipeline based solution of course does not work for Shipwright, given we only leverage TaskRun with tekton right now. We are considering CustomTasks for some integration with various external components with shipwright builds, we have no plans at this time to move off of a TaskRun only approach for the basic build strategy - build - build run flow.

So at this time, possible solutions here do not include IMO direct upstream tekton exploitation, and only include

We do have a burgeoning blog post set that describes third party integrations, like what was recently done for leveraging OPA gatekeeper. So depending on how much this comes up as consumption of shipwright increases, firming up and documenting how to achieve desired levels of serial or parallel execution by what you define the existing build strategy / build / buildrun framework feels like a logical first step before taking on implementing something.

But of course the folks subscribed here chime in and discuss as you desire :-)