Provide tooling for automated Kyma lifecycle and subscription management

[varbanv]

Description

Users should be able to fully automate Kyma related tasks in their CI/CD pipelines. This starts with Kyma instance provisioning, includes deployment of necessary artifacts and test executions, and ends with Kyma instance removal and associated service instance cleanup. All of this should not require much additional effort from the end user and should be as easy as possible to configure.

Context

Problem

Right now, users can automate the Kyma instance creation using the new Terraform provider, the cloud orchestrator tools, or the btpcli directly, however, the next step of accessing the cluster requires user interaction.

And finally, there are a number of scenarios that could prevent a cluster from being deleted and would require user interaction to complete. While the second and third problems can be solved with some coding and additional resources, we want to provide ease-of-use in order to improve the perception and adoption of Kyma.

Benefits

For customers:

• speed up development and release cycles
• implement best practices around CI/CD
• reduce configuration drift
• reduce dependency on long-living Kyma instances
• reduce cost by only using Kyma instances when needed Example usecase scenario that should be possible to run in automated way

For us:

• increase adoption
• enable customers to treat Kyma instances as disposable assets and reduce risk
• enable users to reduce cost

Proposed solution

Design and implement a new set of kyma CLI commands that helps developers who use managed kyma runtimes within BTP ecosystem with development of their CAP applications. CLI commands should be designed with automation flow in mind (No user context should be required),

The commands should cover:

• automated lifecycle management of kyma runtime lifecycle (provisioning, deprovisioning)
• getting access to freshly created cluster
• attaching BTP service instances to kyma runtimes via service manager
• eliminating manual actions (i.e mapping hana instance, establishing trust for custom IAS tenant, etc..)

Acceptance criteria

• [x] Users can create a Kyma instance and access it without any user interaction.
  • [ ] ~https://github.com/kyma-project/kyma/issues/18305~
  • [x] https://github.com/kyma-project/cli/issues/2093
  • [x] https://github.com/kyma-project/cli/issues/2115
• [x] Users can completely remove an existing Kyma instance without any user interaction, even if there are modules "stuck" in deletion.
  • [x] https://github.com/kyma-project/infrastructure-manager/issues/103 - KIM dependant
  • otherwise, this could be done via btp cli subaccount force deletionsubaccount
• [ ] Documentation and guides that cover the full end-to-end CI/CD scenario
  • https://help.sap.com/docs/btp/btp-developers-guide-internal/terraform-module-for-kyma?locale=en-US&state=DRAFT&version=Internal

Attachments

Idea for assisted kyma provisioning

[Disper]

What would be the exact/examples of scenarios around provisioning?

[kwiatekus]

Similar request to log into kyma in headless mode https://github.tools.sap/kyma/backlog/issues/2518

[kwiatekus]

https://github.tools.sap/kyma/backlog/issues/2660#issuecomment-2104608

[pbochynski]

One possible solution is: https://github.com/kyma-project/kyma/issues/18305

[kwiatekus]

What would be the exact/examples of scenarios around provisioning?

@Disper Additional config decribing a "system issuer" should be collected from user and sent to KEB. Further on, provisioner uses this data to deploy the OICD config object and enable the shoot-oidc-service extension. https://github.tools.sap/kyma/backlog/issues/2660#issuecomment-3658944

Provisioner should add cluster role binding to principal represented by tokens issued by the system issuer

[kwiatekus]

Provisioning kyma runtime via BTP CLI https://blogs.sap.com/2022/02/24/creating-sap-btp-kyma-runtime-via-the-sap-btp-cli/

Cloud orchestrator: https://pages.github.tools.sap/cloud-orchestration/

[kwiatekus]

Another approach would be to implement OSB's create_binding API on the KEB side, so that after kyma instance is created one could immediately create binding and use the binding data to access kyma runtime. This has strong benefits:

• complies with OSB specification and with UX around other BTP services (i.e create instance of hana, create binding to access it)
• doesn't require additional OIDC config for simple use cases for short-living disposable kyma runtime (exactly for CI/CD flows)

[kwiatekus]

For users to be able to deploy their code w/o additional subscription (in the DEV mode) we aim to separate deocker-regisrty into separate DEV module to be used also outside of serverless usage https://github.com/kyma-project/kyma/issues/18555#issuecomment-2068633912

[kwiatekus]

As of today, the new prototype commands (developed in https://github.com/kyma-project/cli/tree/v3) allow to

• provision managed kyma runtime within SAP BTP subaccount
• create hana db instance in an automated way (incl. binding and mapping to kyma).

• create references to shared instances

  Getting access to new kyma runtime vi kyma CLI (in 100% automated scenario) will be possible after implementig https://github.com/kyma-project/kyma/issues/18305

[kwiatekus]

Customizable OIDC configurations (required for this epic) are being discussed within https://github.com/kyma-project/infrastructure-manager/issues/134

[kwiatekus]

We started working on cli part:

• kyma cli command that generates kubeconfig for CI/CD workflow https://github.com/kyma-project/cli/issues/2093
• enhancing cli provisioning command to onclude extra oidc config https://github.com/kyma-project/cli/issues/2115 (dependency to KEB https://github.com/kyma-project/kyma-environment-broker/issues/423)
• kyma cli command that helps creating a serviceaccount based kubeconfig https://github.com/kyma-project/cli/issues/2036

ETA - on track

[kwiatekus]

Currently @tobiscr and the team is planning a migration from provisioner to KIM. To start initial testing of the tooling (provisioning and getting access in automated scenario) it would be necessary to switch provisioning of new clusters via kyma infrastructure manager.

[tobiscr]

Plan is to have a working KIM version on DEV latest on end of July 2024

[kwiatekus]

We have synced with @tobiscr : The latest estimate for KIM delivery on prod is end of August 2024.

We could start testing the e2e provisioning (with additional oidc) via kyma cli on DEV only after necessary integration of KEB to KIM is implemented on DEV. @kyma-project/gopher PTAL https://github.com/kyma-project/kyma-environment-broker/issues/423#issuecomment-2194680153

[kwiatekus]

Status update 15.07.2024

• we have planned additional work related to building the kyma cli binary using SAP hyperspace - to ensure compliant binary. (@kyma-project/otters ). We will try out the template for pipelines designed for go code and hope 🤞 there will be no complications.
• the work on KIM side is on going (@kyma-project/framefrog )
• the work on KEB side (changing schema for provisioning params) was refined and planned (@kyma-project/gopher )

[kwiatekus]

Status update 29.07.2024

• @ptesny has came up with (and documented) a recipe how to use custom SAP IAS tenant for automated BTP platform resources creation (incl kyma runtime) and getting access to the newly provisioned cluster in automated scenario
• @otters are working on a template for CI pipeline for automated application integration tests that run on automatically created kyma runtime in btp environment
• @otters are aiming to use btp terraform provider for managing btp resources in creating test resources

[kwiatekus]

Status update 26.08.2024

@kyma-project/otters team continue working on the example repository that contains:

• [x] *.tf (terraform manifests) for all necessary btp resources that need to be provisioned for an automated test scenario
• [x] make targets for provisioning resources, building and deploying sample application, testing sample application
• [x] definition for ADO pipeline (integrated with hyperspace tooling)

Once the pipeline is in place we want to document the sample repo as a working recipe for managing kyma lifecycle in the CI scenario. We estimate that we need extra week.

New delivery date: 06.09.2024

[kwiatekus]

Status update 03.09.2024

• a sample demonstration hyperspace pipeline running integration test of sample app is ready.
• (in progress) providing a dedicated reusable terraform module (not a kyma module!) - https://github.com/kyma-project/terraform-module/issues/1

New delivery date: 13.09.2024

[kwiatekus]

status update 09.09.2024

• demo hyperspace pipeline is using extracted terraform module
• polishing new terraform module :
  • stability and automated CI
  • usage examples

[kwiatekus]

Status update 04.10.2024

Working with technical writers on the documentation (as part of btp development guides) https://help.sap.com/docs/btp/btp-developers-guide-internal/terraform-module-for-kyma?locale=en-US&state=DRAFT&version=Internal

[kwiatekus]

Status update 21.10.2024

• [x] public documentation - simply documenting the usage of tf module
• [ ] internal documentation (in progress)

[kwiatekus]

Status update 04.11.2024

• [x] added section about sap btp terraform provider into internal documentation
• [ ] documentation on help portal is in review