Replace current SecretDefinitions by ExternalSecrets

We are in the process of switching from SecretDefinitons (from secrets-manager) to ExternalSecrets (external-secrets operator).

We need to update our saas-operator to create this new ExternalSecrets custom resources instead of current SecretDefinitons.

We will need to update current CRDs by adding to the fromVault spec:

Optional cluster-store (default value vault-mgmt, so no need to update current CRs with this new field)
Optional refreshInterval (default value 60s, so no need to update current CRs with this new field)
Removing from the vault path field the string secrets/data/ if set (not needed on ExternalSecrets), so no need to update current CRs with this existing field That way current CRs will still work and change will be transparant.

Example, from:

apiVersion: secrets-manager.tuenti.io/v1alpha1
kind: SecretDefinition
metadata:
  name: system-database-seed
spec:
  name: system-database-seed
  type: Opaque
  keysMap:
    DB_NAME:
      path: secret/data/kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      key: DB_NAME
    DB_USER:
      path: secret/data/kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      key: DB_USER
    DB_PASSWORD:
      path: secret/data/kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      key: DB_PASSWORD
    URL:
      path: secret/data/kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      key: URL

To:

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: system-database-seed
spec:
  refreshInterval: 60s
  secretStoreRef:
    name: vault-mgmt
    kind: ClusterSecretStore
  target:
    name: system-database-seed
  data:
  - secretKey: DB_NAME
    remoteRef:
      key: kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      property: DB_NAME
  - secretKey: DB_USER
    remoteRef:
      key: kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      property: DB_USER
  - secretKey: DB_PASSWORD
    remoteRef:
      key: kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      property: DB_PASSWORD
  - secretKey: URL
    remoteRef:
      key: kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      property: URL

As the target secrets will remain the same (the secret.spec.data won't change, so won't be triggered any deployment), to avoid the target secrets being removed temporarily and recreated by external-secrets-operator, it will be needed to:

Set secrets-manager operator to 0 pods
Deploy new saas-operator version
ExternalSecret will be created by saas-operator, and will adopt current target secret without doing changes on the secrets data (if it is configured correctly), which is the one used by saas-operator hashes:
- It will add owner references to target secret (you can check it on argo now, an ExternalSecret resource which creates a Secret resource
- Target secret labels will be merged with ExternalSecrets propagated labels and current ones from secrets-manager (last updated date, and updatedBy secrets-manager)
SecretDefinition won't be deleted, you need to manually edit each SecretDefinition by removing the metadata.finalizers field, then the SecretDefinition can be deleted
Delete manually secret-manager labels last updated date and updatedBy secrets-manager from target secrets
Set secrets-manager operator to 1 pod
Target secrets won't be updated anymore by secrets-manager

3scale-ops / saas-operator

Replace current SecretDefinitions by ExternalSecrets #140