Cutover plan for rollout of ACM including Compass Runtime Agent

[tobiscr]

Description

The following plan describes the rollout of the ACM module which includes also the Compass Runtime Agent:

• [x] Plan productive rollout @tobiscr
  • [x] Agree with @zhoujing2022 on the rollout date of ACM (this will give green light for deletion of reconciler)
  • [x] Inform SRE about the rollout and the removal of Compass Runtime Agent
  • [x] Microdelivery Issue tasks
• [x] Prepare migration: implement bash script/application to enable ACM for all existing SKRs @mvshao
  • [x] Prepare script to count how many Kyma clusters needs ACM module to be enabled @koala7659
  • [x] Ensure migration happens only on SKRs if Application Type CR exists on cluster. @mvshao
  • [x] Make sure ACM is enabled for all SKRs which have Application Type CR in place. @mvshao
  • [x] https://github.com/kyma-project/application-connector-manager/issues/231
• [x] Test the RC release on Canary
  • [x] Test migration application @mvshao
  • [x] Run basic tests: @koala7659
    • [x] Test scenario 1: update of exiting ACM with migration (when it was installed by old reconciler) @koala7659
    • [x] Test scenario 2: installation of ACM on new system @koala7659
    • [x] Test scenario 2: Uninstallation @koala7659
• [x] Release module to Experimental channel and test Commerce integration MAIN TESTING SCENARIO
  • [x] Create Release Candidate and publish in Experimental channel @m00g3n
  • [x] Run integration tests with Commerce: @koala7659
    • [x] Test scenario 1: update of exiting ACM with migration (when it was installed by old reconciler)
      • [x] Disable cluster reconciliation with kcp tool
      • [x] Upgrade module to Experimental release
      • [x] Test Integration with EC
    • [x] Test scenario 2: installation of ACM on a new system - you can upgrade to experimental and remove the module
      • [x] Disable cluster reconciliation with kcp tool
      • [x] Upgrade module to Experimental release
      • [x] Disable module - to cleanup cluster
      • [x] Install on module Experimental release
• [x] Release Application Connector Module in Fast & Regular channels
  • [x] PR to module-manifest repo for FAST Channel @koala7659
  • [x] PR to module-manifest repo for REGULAR Channel @koala7659
  • [x] PR to disable the Compass-Runtime-Agent in mothership reconciler on STAGE and PROD (See https://github.com/kyma-project/control-plane/blob/6846f00a4de9ab02cef877fa5b3800bbf21365db/resources/kcp/values.yaml#L140) @m00g3n
  • [x] Issue change request for SRE to run migration application on SKR STAGE @koala7659
  • [x] Issue change request for SRE to run migration script on SKR PROD @koala7659
• [x] Documentation tasks @grego952
  • [x] Testing strategy
  • [x] Troubleshooting guide
  • [x] What's New note <- user can't register a system until module is enabled @koala7659 @grego952
  • [x] Doc remarks/update <-@koala7659 @grego952
  • [x] Release notes <-@koala7659 @grego952

[tobiscr]

@janmedrek - please see our cutover plan. WE will get in touch with @zhoujing2022 to come up with a date for the possible reconciler shutdown.

[koala7659]

Script to estimate how many Application Connectors modules are used:

set -e

echo "Running Application Connector check script"

if ! kubectl get crd applicationconnectors.operator.kyma-project.io &>/dev/null; then
    echo "Application Connector CRD is missing on the cluster - exiting"
    exit 0
fi

echo "Application Connector CRD exists on the cluster"

if ! kubectl get applicationconnectors.operator.kyma-project.io -n kyma-system applicationconnector-sample &>/dev/null; then
    echo "Application Connector CR is missing on the cluster - exiting"
    exit 0
fi

CRState=$(kubectl get applicationconnectors.operator.kyma-project.io -n kyma-system applicationconnector-sample -oyaml | yq .status.state)

echo "Application Connector CR exists on the cluster and is in state ${CRState}"

if ! kubectl get deploy -n kyma-system application-connector-controller-manager &>/dev/null; then
    echo "Application Connector Manager deployment is missing on the cluster - exiting"
    exit 0
fi

echo "Application Connector Controller Manager deployment is running on the cluster!"

if ! kubectl get crd applications.applicationconnector.kyma-project.io &>/dev/null; then
    echo "Application CRD is missing on the cluster - exiting"
    exit 0
fi

echo "Application CRD exists on the cluster"

applications=$(kubectl get applications.applicationconnector.kyma-project.io --ignore-not-found -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')

if [ -z "${applications}" ]; then
    echo "No applications are present on the cluster - no external system is connected for the integration"
fi

for application in $applications; do
    echo "Application found $application"
done

[Disper]

• [x] @koala7659 & @m00g3n trying to figure out why the eventing is not working in the latest version, what is causing the problem is not yet known
• [x] release the fix. New Release 1.1.3
• [x] re-test aiming to be ready for end of the week

[koala7659]

Current Release Candidate 1.1.3

[koala7659]

Script to verify state of Application Connector module after upgrade to version 1.1.3:

set -e

echo "Running Application Connector check script after migration to version 1.1.3"

if ! kubectl get crd applicationconnectors.operator.kyma-project.io &>/dev/null; then
    echo "Application Connector CRD is missing on the cluster - Module is not enabled. Exiting"
    exit 0
fi

echo "Application Connector CRD exists on the cluster"

if ! kubectl get applicationconnectors.operator.kyma-project.io -n kyma-system applicationconnector-sample &>/dev/null; then
    echo "Warning! Application Connector CR is missing on the cluster - Module is probably being installed or deleted. Exiting"
    exit 1
fi

CRState=$(kubectl get applicationconnectors.operator.kyma-project.io -n kyma-system applicationconnector-sample -oyaml | yq .status.state)

echo "CR state is: ${CRState}"

if [ "${CRState}" != "Ready" ]; then
  echo "Warning! Application Connector CR is in state ${CRState} - exiting"
  exit 1
fi

if ! kubectl get deploy -n kyma-system application-connector-controller-manager &>/dev/null; then
    echo "Error! Application Connector Manager deployment is missing on the cluster - exiting"
    exit 1
fi

#checking CRDs

if ! kubectl get crd compassconnections.compass.kyma-project.io &>/dev/null; then
    echo "Error! Compass Connection CRD is missing on the cluster - Module is not enabled. Exiting"
    exit 1
fi

if ! kubectl get crd applications.applicationconnector.kyma-project.io &>/dev/null; then
    echo "Error! Application CRD is missing on the cluster - exiting"
    exit 1
fi

#check compassconnection state
ComConState=$(kubectl get compassconnections compass-connection -oyaml --ignore-not-found | yq .status.connectionState)

echo "Compass Connection state is: ${ComConState}"

if [ "${ComConState}" == "Connected" ]; then
  echo "Warning! Compass Connection is Connected but is waiting for application synchronisation - exiting"
  exit 0
fi

if [ "${ComConState}" != "Synchronized" ]; then
  echo "Warning! Compass Connection is in state ${ComConState} - exiting"
  exit 1
fi

# echo "Application CRD exists on the cluster"
applications=$(kubectl get applications.applicationconnector.kyma-project.io --ignore-not-found -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')
if [ -z "${applications}" ]; then
    echo "No applications are present on the cluster - no external system is connected for integration. Module can be disabled."
    exit 0
fi

echo "Some applications are connected to the cluster. Module must remain enabled."

[koala7659]

4 clusters on prod after migration cannot synchronise applications with Compass. Issue to investigate: https://github.com/kyma-project/application-connector-manager/issues/259

[koala7659]

Future release 1.1.4 https://github.com/kyma-project/application-connector-manager/releases/tag/1.1.4

[koala7659]

Release is completed. Application-Connector module version 1.1.3 is available on both fast and regular channels