[betaEnabled] Support Beta flag in KEB and Kyma CR

[PK85]

Description

The KEB needs to store “Used for production” and “Enabled beta features” information during Kyma provisioning (Only provisioning).

Reason

This feature is of high importance as it directly impacts the ability of customers to utilize beta modules effectively within Kyma. Being able to discern user preferences for beta features during Subaccount creation will enable better alignment with customer needs and preferences.

AC

• [x] 0) do local POC with all proper working calla to CIS APIs and present to the team, then next points: - [ ] 1) When KEB is called with ers_context structure we need to take subaccountID and find “Enabled beta features” information in the Account Service API. Then KEB persists that in the DB and labels Kyma CR during Kyma provisioning [EDIT: not valid any more]
• [x] 2) Be able to check when beta flag has been change on our side
• [x] 3) figure out how to test that new feature
• [x] 4) Update all Runtimes created in the past that have missing beta information.
  • [x] 4.1) run it at least once
• [x] 5) Provide documentation about new integration

Info

• Docs about Cloud Management Service (CIS) APIs:
  • https://accounts-service.cfapps.eu10.hana.ondemand.com/api?urls.primaryName=Accounts-Service-Api$cis-system&scope=cis-system#/g_Accounts%20APIs/getSubaccount_1
  • https://help.sap.com/docs/btp/sap-business-technology-platform/sap-cloud-management-service-service-plans

[PK85]

We encountered that "Enable beta features" could be selected to "true" after subaccount creation. This changes the way we can implement this feature.

In that case we do not need to call CIS APIs during provisioning. We can check events let say every 1-2 hours and check betaEnabled from event itself.

To get information about betaEnabled for older Kyma instances the only way is to call Account Service APIs and check subaccount. But right now we receive 403.

Comments:

• look for "Subaccount_Update" event type from Event Service API, then if "subaccountId" matches any of our Runtimes(right now it should be max exactly 1) do a call to AccountService API to check is beta has changed. Answer: in this scenario calling Account Service API is not required, we see that the event's payload contains betaEnabled.
• Cloud Management Service does not contain system-viewer plan. Only system-basic and xrs
• When calling Account Service API we got 403, looks like we need proper scope: Required scope: $XSAPPNAME.global-account.subaccount.read

[jaroslaw-pieszka]

Having subaccount with Cloud Management Service (cis) and system-basic plan you can get Subaccount_Update event when betaEnabled or usedForProduction has changed. However in the event payload there is not information about usedForProduction even if change of this value triggered appropriate event.

betaEnabled and usedForProduction checked in cockpit - event generated:

"events": [
        {
                "id": 25290915,
                "actionTime": 1706600916546,
                "creationTime": 1706600917056,
                "details": {
                        "description": "Subaccount updated.",
                        "guid": "e6c2c9d4-49d9-4da7-92be-00e24fe02f51",
                        "technicalName": "e6c2c9d4-49d9-4da7-92be-00e24fe02f51",
                        "parentGuid": "156d610b-eda1-44ab-a093-4977fca4e43e",
                        "displayName": "jp-test",
                        "subaccountDescription": null,
                        "region": "eu12",
                        "subdomain": "jp-test-pn9upk1f",
                        "betaEnabled": true
                },
                "globalAccountGUID": "156d610b-eda1-44ab-a093-4977fca4e43e",
                "entityId": "e6c2c9d4-49d9-4da7-92be-00e24fe02f51",
                "entityType": "Subaccount",
                "eventOrigin": "accounts-service",
                "eventType": "Subaccount_Update"
        },

usedForProduction unchecked in cockpit - event generated:

"events": [
        {
                "id": 25290926,
                "actionTime": 1706601042984,
                "creationTime": 1706601044047,
                "details": {
                        "description": "Subaccount updated.",
                        "guid": "e6c2c9d4-49d9-4da7-92be-00e24fe02f51",
                        "technicalName": "e6c2c9d4-49d9-4da7-92be-00e24fe02f51",
                        "parentGuid": "156d610b-eda1-44ab-a093-4977fca4e43e",
                        "displayName": "jp-test",
                        "subaccountDescription": null,
                        "region": "eu12",
                        "subdomain": "jp-test-pn9upk1f",
                        "betaEnabled": true
                },
                "globalAccountGUID": "156d610b-eda1-44ab-a093-4977fca4e43e",
                "entityId": "e6c2c9d4-49d9-4da7-92be-00e24fe02f51",
                "entityType": "Subaccount",
                "eventOrigin": "accounts-service",
                "eventType": "Subaccount_Update"
},

[jaroslaw-pieszka]

Meeting minutes:

1. Once a day we do a full sync i.e. for all subaccounts we fetch current state of betaEnabled, usedForProduction using account (technical) endpoint
2. During remaining time we do incremental sync i.e. we fetch events (Subaccount_Update, Subaccount_Create) and check if above-mentioned values were changed of set in the case of create event. We do not get value for usedForProduction in event details - to be clarified with CIS team if this is to be changed
3. We initiate incremental sync every 15 minutes (for example) with overlapping window (20 minutes) to minimize risk of missing any relevant event.
4. Times, intervals should be configurable,
5. Persist (in the storage) the information about desired and current state of kyma resources in regard to these two values [edit: this is disputable, possibly we can recreate the information after every restart and then for given period),
6. Persist state of sliding event window,
7. Metrics are to be introduced,
8. Handle rate limiting (accounts and events endpoints),
9. Handle errors while updating kyma resources (retry later on),
10. Check with jellyfish the exact labels to be used, make sure no such labels are there - we should start from the scratch,
11. It would be nice to have single credentials for events and accounts (at the time being we have to sets),

[jaroslaw-pieszka]

Ad 5. Persistence is not necessary, if the apps runs permanently in memory, we can keep state there and rebuild it after restart by full sync with CIS, KCP and database.

Ad 6. Not needed. Just after app start we do a full sync and later on periodic event scan with the sliding window.

Ad 7. Not in MVC0.

[jaroslaw-pieszka]

We requested on Jan, 31st adding "usedForProduction" to Event details. Answer from Serhan:

This will be handled in T02 (staring next week) in CIS dev team: Add usedForProduction flag value to the > Subaccount_Create/Update/Delete event payloads So you can expect it with the release of T2402

[jaroslaw-pieszka]

• [x] store states in DB - since eventually we want to keep state in DB as well (usedForProduction in DB only, betaEnabled in DB and Kyma resources)
• [ ] get credentials (get secret created with credentials) allowing to call accounts technical API (PK is supposed to get it done)
• [x] create all needed configurations in values.yaml and management control plane
• [x] create deployment
• [x] add logic updating resources https://github.com/orgs/kyma-project/projects/38/views/1?pane=issue&itemId=53511051
• [x] add logic updating DB
• [ ] add logic to fetch usedForProduction with additional accounts call (on feature flag that could be switched off when usedForProduction present in event details - see https://jira.tools.sap/browse/SAPBTPCFS-10256 for details about availability of usedForProduction in events details.
• [x] align with jellyfish team the label name.
• [x] implement unit tests
• [x] implement e2e tests
• [ ] test configuration and deployment
• [ ] test run (dry and wet)
• [ ] create documentation

[jaroslaw-pieszka]

Meeting minutes:

• states (settings of subaccounts of interest i.e. used for SKR creation) should be persisted in database
• worker queue containing subaccounts for which resources should be updated, should be persisted in database, on applications start we do not build all states and queue from scratch but retrieve it from database.
• application should withstand unavailbility of CIS services (events and accounts)
• metrics are crucial

[szwedm]

CIS fake server for tests: https://github.com/kyma-project/kyma-environment-broker/pull/589

[jaroslaw-pieszka]

PR for review: https://github.com/kyma-project/kyma-environment-broker/pull/460

[jaroslaw-pieszka]

App deployed on dev. Deployment to stage and prod and monitoring setup will be done in scope of separate issues.