KEB: MIgrate from provisioner to Kyma Infrastructure Manager

[PK85]

Acceptance Criteria

Phase 1: (driving system Provisioner for creation, Runtime CR spec is not passed to gardener, Runtime CR status is already present)

• [x] 0) Import (if no dependencies added)/copy CRD structures from KIM - https://github.com/kyma-project/kyma-environment-broker/issues/905 - for the time being we decided that we will handle this later on.
• [x] 1) use preview plan to use provisioner and KIM - currently configurable for any plan of preference
• [x] 2) create Runtime CR ~and create GardenerCluster~ (for later: do not create GardenerCluster CR, just watch a secret by convention name)
  • [x] 2.1 Do not create Gardener Cluster, KIM Team decided to manage it on their own: https://github.com/kyma-project/infrastructure-manager/issues/282
• [x] 3) support update operation which call Provisioner and patch Runtime CR #1068
• [x] 4) support deprovisioning operation which call Provisioner and delete Runtime CR
• [x] 5) support suspension/expiration operation which call Provisioner and delete Runtime CR #1069
• [x] 6) support upgrade cluster operation which call Provisioner and delete Runtime CR (this one we need to double check, cause with Runtime CR SRE will upgrade cluster just by modifying Runtime CR directly) #1072 Upgrade would be handled by SRE
• [x] 7) check status from Runtime CR, maybe try to error shoot creation and check a status syntax
• [x] 8) update kcp CLI and runtimes endpoint to support instances not known to provisioner (KIM driven) and to support fetching Runtime CR spec. #1070
• [x] 9) adjust SKR tests to use Runtime config or Gardener config returned by kcp CLI call (extension in #1070 are needed) #1077
• [x] 10) refactor to reduce technological debt.

Expected result from phase 1:

• we figured out all not working scenarios, sytax etc
• provisioning and update works, and deprovisioning
• KIM controller works as expected and status from provisioner and KIM should be generally the same (READY, ERROR ), shoot from provisioner and shoot CR from KIM are the same
• skr-preview tests are green - https://github.tools.sap/kyma/gophers-pipelines/actions/workflows/skr-preview-dev.yaml
• we are ready to go to phase2

Phase 2: (driving system KIM for creation, Provioner spec is not passed to gardener, Runtime CR status is already present)

• [x] 0) KEB preview is checking label on Runtime CR (old/new) and then check provisioner status if old, and Runtime CR if new one (on DEV preview works with KIM only, otehr plans create runtimeCR but usese provisioner, flag is taken into account)
• [x] 1) Preview plan check status from Runtime CR (maybe feature flag for Preview plan only that new hidden input parameter is availabe by which we can decide from where we check status)
• [ ] 2) Frogs run migration, all existing SKRs will be represented by RuntimeCR too
• [ ] 3) Enable preview implementation in all KEB plans with feature flag ON on DEV (stage and prod still OFF) - RuntimeCR is till in view mode only on KIM side

Expected result from phase 2:

• by Runtime CR new label we can switch to KIM or/and revert to Provisioner and everything should work
• recreating all Runtime CT by frogs tested
• KIM became the driving system we can go to phase 3

Phase 3:

• [ ] 1) Remove call to provisioner from all plans and operations. KEB works only with Runtime CR

Expected result from phase 3:

• KEB is using only Runitime CR
• Provisioner is undeployed from all environmnets

[piotrmiskiewicz]

Technical detail: Do not use InputCreator for creating Runtime CR. All information we need to create the CR we have in the Operationb and Provisioning Parameters.

[jaroslaw-pieszka]

@kyma-project/framefrog We need to clarify some fields (presence, values):

• spec.shoot.provider.workers.name - described as optional, but required - what name should we provide, will we select workers group later on by name or index (assigning parameters to workers[0] or worker of specific name)
• LicenceType - it is the CRD spec, do you use it, is it optional, should it be in the spec or should it be a label
• Networking.Filter.Egress.Enabled - value based on LicenceType - is the value the same as ShootNetworkingFilterDisabled in provisioner contract?
• euAccess - present in provisioner contract, not in Runtime CRD - should we add it as a label or should the spec be extended?
• ExposureClassName- we passed it for sap-converged-cloud - should we somehow support it?
• EnableKubernetesVersionAutoUpdate - was in the contract with provisioner - not used/forgotten?
• EnableMachineImageVersionAutoUpdate - was in the contract with provisioner - not used/forgotten?

Provider specific fields (sent to provisioner):

• Azure: VnetCidr,EnableNatGateway, IdleConnectionTimeoutMinutes)
• AWS: VpcCidr, EnableIMDSv2
• sap-converged-cloud: FloatingPoolName, CloudProfileName, LoadBalancerProvider

[piotrmiskiewicz]

@kyma-project/framefrog I have a questions regarding a time of migration and the update. Let's imagine a scenario:

1. SKR is created using Provisioner.
2. Migration to KIM is executed (runtime CR is created).
3. The customer updates the SKR (for example changes the machine type)
4. How KEB should know if the Provisioner should be called with an update mutation or not? Runtime CR will always be updated if exists.

[akgalwas]

@kyma-project/framefrog We need to clarify some fields (presence, values):

• spec.shoot.provider.workers.name - described as optional, but required - what name should we provide, will we select workers group later on by name or index (assigning parameters to workers[0] or worker of specific name)
• LicenceType - it is the CRD spec, do you use it, is it optional, should it be in the spec or should it be a label
• Networking.Filter.Egress.Enabled - value based on LicenceType - is the value the same as ShootNetworkingFilterDisabled in provisioner contract?
• euAccess - present in provisioner contract, not in Runtime CRD - should we add it as a label or should the spec be extended?
• ExposureClassName- we passed it for sap-converged-cloud - should we somehow support it?
• EnableKubernetesVersionAutoUpdate - was in the contract with provisioner - not used/forgotten?
• EnableMachineImageVersionAutoUpdate - was in the contract with provisioner - not used/forgotten?

Provider specific fields (sent to provisioner):

• Azure: VnetCidr,EnableNatGateway, IdleConnectionTimeoutMinutes)
• AWS: VpcCidr, EnableIMDSv2
• sap-converged-cloud: FloatingPoolName, CloudProfileName, LoadBalancerProvider

@jaroslaw-pieszka Please see my comments below

We agreed the following:

• KIM will take over some functionalities related to infrastructure
• Knowledge that is related to plans should stay in KEB, everything else can be moved to KIM

For more details on the components responsibilities please see this ADR document.

Answers:

• spec.shoot.provider.workers.name you can use any string, Provisioner has hardcoded cpu-worker-0
• LicenceType is not required ; when passed in the CR it will be used for setting infrastructuremanager.kyma-project.io/licence-type annotation ; basically it is equivalent to licenceType field in GraphQL
• Networking.Filter.Egress.Enabled is a negation of Provisioner's ShootNetworkingFilterDisabled
• KIM handles euAccess annotation (please see this code)
• KIM handles ExposureClassName (please see this code)
• Both EnableKubernetesVersionAutoUpdate and EnableMachineImageVersionAutoUpdate is handled by KIM (please see the code and configuration
• KIM handles details of the Provider Specific Config (please see the code for aws, azure, gcp, openstack)