search

kyma-project / lifecycle-manager

Controller that manages the lifecycle of Kyma Modules in your cluster.
http://kyma-project.io
Apache License 2.0
9 stars 30 forks source link

Module catalogue improvements #1472

Open janmedrek opened 2 months ago

janmedrek commented 2 months ago

Description

Module catalogue (in this case - collection of Module Templates setup) should be adapted to the new requirements:

As for the implementation options, there are two main ones:

Reasons

We should adapt to the changing requirements and Community Modules concept.

Acceptance Criteria

Open Questions

jeremyharisch commented 1 month ago

Acceptance Criteria for EPIC:

Timebox: 2 Days

Sub Issues:

Tomasz-Smelcerz-SAP commented 1 month ago

I think we should use K8s explicit types and avoid obscure solutions (like mounted ConfigMaps/Secrets) for the Module Catalogue. That's why I propose to have the Module Catalogue defined as: Module Catalogue is a collection of Module Templates in the cluster the Lifecycle-Manger is deployed.

Tomasz-Smelcerz-SAP commented 1 month ago

Requirement 1:

A given module should be available in multiple different versions, users should be able to enable a specific version in their cluster. This requirement does not contradict the channel concept. It should still be possible to "subscribe" to a given channel (which will be one of the versions marked with that channel).

Use case 1: User wants to enable a specific version of a module in their cluster.

[...]
- modules:
  - name: my-module
    version: 1.0.0
[...]

Use case 2: User wants to enable a specific channel of a module in their cluster.

[...]
- modules:
  - name: my-module
    channel: fast
[...]

Analysis:

This requirement implies there are many versions of the given module to choose from. It is best modelled with many ModuleTemplate CRs - one for each version of the module. It's better to have multiple ModuleTemplates (one for a version) instead of one "big" ModuleTemplate for all versions, because the ModuleTemplate should be immutable. The benefits of immutability is, for example, much easier caching of the data. If we have a single ModuleTemplate CR for a given module that describes all of it's versions, then when a new version appears this ModuleTemplate CR must be edited (mutated). That's why it's better to have a single ModuleTemplate CR for a given version of a given module.

Another requirement is that versions may "belong to" a set of channels, so there is a mapping between versions and channels. A version may be assigned to a (possibly empty) set of channels. It seems reasonable then to add this mapping to the ModuleTemplate CR.

API Changes:

Implementation details:

ModuleTemplate:

An example of current ModuleTemplate CR type (version is an annotation):

kind: ModuleTemplate
metadata:
  name: template-operator-regular
  namespace: kcp-system
  labels:
    "operator.kyma-project.io/module-name": "template-operator"
  annotations:
    "operator.kyma-project.io/doc-url": "https://kyma-project.io"
    "operator.kyma-project.io/is-cluster-scoped": "false"
    "operator.kyma-project.io/module-version": "0.1.2" 
spec:
  channel: regular
  mandatory: false 
  data:
  [...]
  descriptor:
  [...]

An example of the "new" ModuleTemplate CR type with the version attribute:

kind: ModuleTemplate
metadata:
  name: template-operator-0-1-2
  namespace: kcp-system
  labels:
    "operator.kyma-project.io/module-name": "template-operator"
  annotations:
    "operator.kyma-project.io/doc-url": "https://kyma-project.io"
    "operator.kyma-project.io/is-cluster-scoped": "false"
spec:
  name: "template-operator"
  channels: ["regular", "fast"]
  version: 0.1.2
  mandatory: false 
  data:
  [...]
  descriptor:
  [...]

In addition I would like to have the module name be an explicit attribute in the ModuleTemplate CR. The label may remain for easier searching, but the module name should be explicit.

Tomasz-Smelcerz-SAP commented 1 month ago

Modified channel/version design proposal:

At this moment we only allow users to select the channel. After the change, the user will also be able to select the version. for that to work, the version must refer to a unique ModuleTemplate.

I propose to make the following changes in the ModuleTemplate API:

Example:

  1. A module version defined in the "regular" channel 

    kind: ModuleTemplate
[...]
spec:
channels: ["regular"]
version: 0.1.1

  2. A module version defined in two channels:

    kind: ModuleTemplate
[...]
spec:
channels: ["regular", "fast"]
version: 0.1.2

  3. A standalone module version (no channel):

    kind: ModuleTemplate
[...]
spec:
channels: []
version: 0.1.3

Channels mutability issues:

What if the team releases a module version as standalone, and later wants to publish it in the fast channel? Similarly - what if the team initially publishes a module version in fast channel and later on decides to publish it also in the regular channel? We have two options here:

  1. Somehow we allow the channels attribute to be modified in the ModuleTemplate
  2. The team must release a new version for that purpose, referring to the same OCM artifact(s)

I opt for option 2. as it simplifies module management - as ModuleTemplates are immutable, it's simpler to distribute/cache them.

Tomasz-Smelcerz-SAP commented 1 month ago

Requirement 4

For each module, there should be a list of "domain CRDs" that this module brings (i.e. Serverless will have Functions, Istio will have Gateways, Virtual Services...) that can be used to see which CRDs correspond to which module.

Option 1: This can be done during the ModuleTemplate creation. We can extends modulectl create module command (not yet existing) to parse the resources in order to find the CRDs. In addition we may allow user to explicitly add some CRDs manually.

The modified ModuleTemplate CR example:

kind: ModuleTemplate
metadata:
  name: kyma-istio
  namespace: kcp-system 
spec:
  version: 1.2.3
  mandatory: false
  managedResources:
  - apiVersion: "networking.istio.io/v1beta1"
    kind: "Gateway"
  - apiVersion: "networking.istio.io/v1beta1"
    kind: "VirtualService"
  [...]

Option 2: The managedResources list is just a part of the module and is stored as a separate layer in the module artifact. The advantage of this approach is that the managedResources is bound to the module definition at the module creation time, and it cannot be easily changed. This eliminates all sorts of errors related to invalid module configuration in the control-plane.

Tomasz-Smelcerz-SAP commented 1 month ago

Requirement 3

For the restricted markets there should be an option to configure the repository from which manifests are fetched (i.e. China market will use the China-restricted repository).

For now there is not enough information provided about the possible scenarios here. But it looks like we can implement this in the Lifecycle-Manager code, without affecting the API. One possible solution would be to add additional metadata to some ModuleTemplate CRs that would define the alternative repository URL. It could be an annotation. The Lifecycle-Manager would then use this metadata instead of the original repository URLs (taken from OCM descriptor) to fetch the relevant manifests.

Note: If all the repository URLs are to be replaced with a single value, this can also be implemented as the CLI argument to the Lifecycle-Manager process. This would then affect ALL Modules in given environment. We can implement both options anyway.

ruanxin commented 1 month ago

Hi @pbochynski ,

We have some questions here, based on the current shipping kcp concept, we are versioning release channel, with the current approach, KLM is possible to use the latest channel version to get a list of latest module version.

However, in order to support the following request:

A given module should be available in multiple different versions, users should be able to enable a specific version in their cluster. This requirement does not contradict the channel concept. It should still be possible to "subscribe" to a given channel (which will be one of the versions marked with that channel).

How it aligned with the channel version then, to support multiple module versions, KLM must know all history versions of the channel first. e.g: channel/fast:1.0.0 contains istio:1.0.0 channel/fast:1.0.1 contains istio:1.1.0 to provide istio 1.0.0 and istio 1.1.0 to user in fast channel, KLM must know OCM component channel/fast have two versions(1.0.0, 1.0.1) exists. Which means KLM have to introduce a process to scan all channel/fast in OCM registry to build a reference between module and channel upfront. Is it necessary? But then, is it better we consider channel as a label attribute to kyma module?

Other questions:

  1. Although here requested to support multiple versions, the version downgrade should still prevented, right?
  2. Does this feature also means we allow user to always stay in an old version without upgrading?
  3. What's the value behind this feature? consider the challenges may increase maintenance efforts and compatibility issue,
    • for example, an old istio version maybe not supported by other newer module which depends on it.
    • and allow user to freely choose different version, especially between major version, it might bring breaking change, especially during the CRD API upgrade. e.g: 1.0.0 contains v1alpha1, in 1.1.0 introduce v1alpha1, v1alpha2, in 1.2.0 deprecated v1alpha1, but this version should make sure handle the version upgrade properly, then in 1.3.0 only keep v1alpha2, if user directly jump from 1.0.0 to 1.3.0, it's not possible, API server will not allow apply the CRD without v1alpha1
pbochynski commented 1 month ago

You can have a look at my PoC where I describe all component versions available in all channels: https://raw.githubusercontent.com/kyma-project/community-modules/main/model.json I use channels as a version attribute there. You do not need to know the history of the channel: you just apply the version for the channel or apply the version selected by the user. As in the current solution, you prevent downgrades based on semantic versioning. The only difference to the current logic is that if the current version is older than the oldest version listed in the shipment you automatically upgrade it (the user can't stay behind). This behavior is the same as what we have now with the regular channel (you can't stay behind regular channel also in the future)

The value of the feature is that you can decide about timing. You can still have a test cluster pinned to the fast channel, but your production channel you will upgrade when your SRE/dev team is available, and customers won't be affected. You can switch your production cluster to the version from the fast channel the next day after you successfully tested it on another cluster. So it is more control for the user. Of course, if you don't upgrade in 2 weeks (or whatever time we still support that version) you will be upgraded automatically.

ruanxin commented 1 month ago

After clarified with @pbochynski:

  1. Mutiple version means user are free to subscribe module within channel or choose one version specifically.
  2. Module team able to publish version to channel or optionally offer some module version which allow user to subscribe with.
  3. Version downgrade is still not supported
  4. During iteration, if the version subscribed by user is not managed by module team (not appears in channel or offered by module team specifically), this module becomes unmanaged.
Tomasz-Smelcerz-SAP commented 1 month ago

@pbochynski Hi Piotr, I have a question related to "managedResources". In your POC this list is common to all module versions. Is there a chance that this list changes from version to version? I think it's a valid use case. What then? The team must release a new module, with a different name?

c-pius commented 2 weeks ago

Regarding the following statement from PB (see https://github.com/kyma-project/lifecycle-manager/issues/1472#issuecomment-2129344858):

You do not need to know the history of the channel: you just apply the version for the channel or apply the version selected by the user. As in the current solution, you prevent downgrades based on semantic versioning. The only difference to the current logic is that if the current version is older than the oldest version listed in the shipment you automatically upgrade it (the user can't stay behind). This behavior is the same as what we have now with the regular channel (you can't stay behind regular channel also in the future)

Not exactly sure what "shipment" means, but I would interpret this as follows:

  • support for a module version may be sunset at a given point in time ("the current version is older than the oldest version listed in the shipment")
  • when it has been sunset, an automatic update is forced on the version

If this understanding is correct, we need to clarify:

EDIT: discussed the above with Xin. Basically, this is already described in point 4. here: https://github.com/kyma-project/lifecycle-manager/issues/1472#issuecomment-2129614726. If the user configured a channel, e.g., regular, the module version will be auto updated with the evolution in the channel. If the user configured a fixed version, and the support for this version is dropped, we are not auto updating it and the module becomes unmanaged.