Implement Version Management for Builders, BuildPacks and Quickstarts

[tdcox]

There is an intrinsic dependency relationship between quickstarts, builders and buildpacks that spans the lifecycle of any customer application code derived from them.

We should assert the following use-cases:

1. Customers should always be able to rebuild an application that they have previously successfully built on the platform, within an agreed time window of support that should be measured in years.

2. It must be possible for the Jenkins-X project to upgrade builders on a regular cadence to mitigate vulnerabilities and add new features.

3. It must be possible for the Jenkins-X project to upgrade buildpacks on a regular cadence to mitigate vulnerabilities and add new features.

4. There must be a migration path for customer codebases to upgrade to later versions of builders and buildpacks (on the basis that this may necessitate changes to their codebase).

5. It must be possible for the Jenkins-X project to deprecate and eventually remove instances of quickstarts, builders and buildpacks at the end of a support window.

Implications:

• Quickstarts, buildpacks and builders must be versioned
• The relationship between related versions of quickstarts, builders and buildpacks must be managed and maintained by the platform
• There is a requirement to support multiple versions of quickstarts, builders and buildpacks in parallel on any given platform instance
• The cadence of upgrades is much shorter than the desired length of support window, therefore there is a challenge to managing multiplying numbers of quickstart, builder and buildpack instances
• Under the current approach, builders contain jx client code and are therefore not immutable and must be regenerated upon each release
• Builders act as a local cache of dependencies on customer instances, to accelerate build activities
• Changes to builders invalid local caches and can introduce unexpected errors to previously working codebases
• Versions of quickstarts, builders and buildpacks may be tied to versions of nodepools on customer K8S instances, especially in MLOps scenarios where customers are using dedicated hardware resources such as GPU or TPU instances.

Drivers:

1. It is currently not possible to upgrade builders and buildpacks without breaking customer code. This has meant that we are not providing timely security patches to core dependencies.

2. It is not possible to complete the transition from Skaffold to Kaniko due to buildpack dependencies.

3. It is currently not possible to support the rapid pace of change to core frameworks in MLOps due to the need to be able to support multiple versions of dependencies in parallel. in production.

Constraints:

• Under the current model, proliferation of builders would have a severely detrimental effect upon the performance of the Jenkins-X release process

It is proposed that we discuss ways to re-architect the current solution to meet the above use cases.

See also https://github.com/jenkins-x/jx/issues/4671

[deanesmith]

Next Steps:

• Collaborate on a new proposal issue
• Establish very descriptive and guidance about the problem that needs to be resolved
• Describe the patterns for the design and what the problem space looks like
• Consider carefully differences in cultures in customer bases (dev's/operators)

Notes: Versioning of buildpacks and quickstarts needs to be established first

• Tack on to current architecture (option 1 - NO)
• Re-architecture for a cleaner solution (option 2)
• Minimize overhead

Best suited for a Labs project to establish patterns

MLOps has physical dependencies on hardware Libraries changing ML frameworks changing

Rework image usage/binaries Existing versioning on quickstarts is only conceptual

[jenkins-x-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://jenkins-x.io/community. /lifecycle stale

[tdcox]

/remove-lifecycle stale

[jenkins-x-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://jenkins-x.io/community. /lifecycle stale

[jenkins-x-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close. Provide feedback via https://jenkins-x.io/community. /lifecycle rotten

[jenkins-x-bot]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten. Provide feedback via https://jenkins-x.io/community. /close

[jenkins-x-bot]

@jenkins-x-bot: Closing this issue.

In response to [this](https://github.com/jenkins-x/enhancements/issues/24#issuecomment-734429399): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. >Provide feedback via https://jenkins-x.io/community. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository.