Pipeline Spec to reduce Stage boilerplate

[jessesuen]

Proposed Feature

When talking with users who may have many Stages in their pipeline, they complain about a lot of repeated portions of the Stage spec that are duplicated across their Stages. Kargo should provide some convenience to avoid this duplication.

Motivation

• Reduce boilerplate when there are few differences between Stages.
• Centralize configuration changes so that settings can be changed once, and then apply to all Stages.

Suggested Implementation

There are many ways we could go about this, and I think we will want to experiment with some ideas.

1. Codegen from a "Pipeline" spec

First thought is to define a Pipeline API spec that generates Stages. This Pipeline would not actually be implemented as a CRD in the cluster, but something that produces the Stage CRDs that would live in the cluster.

2. Pipeline CRD

Second option is to have a formal Pipeline CRD that is deployed, in lieu of or to complement Stages. This would require an actual controller running to handle the Stage lifecycle.

3. Template CRD

Another option is to allow Stages to reference something like a "template", in order to reduce the boilerplate. Templates would need to live in the cluster since they are referenceable.

[hiddeco]

This Pipeline would not actually be implemented as a CRD in the cluster, but something that produces the Stage CRDs that would live in the cluster.

While this would probably be the simplest approach, I think this would potentially create new problems. As it would require specific tooling (e.g. kargo) to generate and synchronize your Stages on a cluster, or have them generated before you commit them to a Git repository. This could create new usability issues, either because it's less straight forward to do it in a GitOps manner, or because you have to remember to generate updates to Stages when you make changes to the Pipeline.

[krancour]

@jessesuen isn't this exactly what we intended StageSet for? https://github.com/akuity/kargo/issues/339

Edit: I'd be fine with it if we called it Pipeline instead of StageSet.

[jessesuen]

Yes, https://github.com/akuity/kargo/issues/339 might be the outcome if we chose to go with option 2. However, when filing https://github.com/akuity/kargo/issues/339, one use case was that StageSets could be coupled with other Stages or even other StageSets. And so StageSets were not necessarily the only top-level thing that needed to be defined. If we capture that use case with Pipelines we should merge the concepts.

Option 1 might be a precursor to 2.

[krancour]

btw... I agree with @hiddeco's POV that not doing this as a CRD adds complications.

My preference would be something like this:

• Support a template StageSpec. For promotion mechanisms, especially, the only variable from Stage to Stage is likely to be derivable from the Stage name.

• Support a map of Stage names to StageSpecs. Any of the top-level fields in each of these StageSpecs can be used to override what's in the template StageSpec. The main uses for this would be:

  • Specifying subscriptions. Which Stages are upstream from which cannot be inferred from Stage name, so it needs to be specified on a Stage by Stage basis.

  • Specifying verification processes. Different processes may be used at each stage. For instance, the things verified in QA and "perf" might be different.

one use case was that StageSets could be coupled with other Stages or even other StageSets

imho, that complicates things enormously. Given that most pipelines have multiple production environments on the far right, and some can have multiple entrypoints on the far left, coupling StageSets together wouldn't be especially simple.