Smart secret management

[eguzki]

https://issues.redhat.com/browse/THREESCALE-7904

Apicast operator will manage related secrets with the following approach:

• Reconcile secret contents (best UX). In place changes of the content of the secret will be notified to the operator, which will rollout apicast deployment.
• The apicast operator does not take ownership of any secret. This allows external management of the secrets with other operators (like sealed secrets)
• The CR referencing the secret (i.e. APIcast CR) will have labels with the ID of the secrets being referenced. The controller will add the secret ID's in the .metadata.labels section of the APIcast CR.
• The controller watches secrets with the custom label apicast.apps.3scale.net/watched-by=apicast. This secret label is not required to be referenced from the APIcast CR. Only when this label is in place, the operator will get notified when the secret content changes (or the secret is created or deleted). If this label is not included in the secret, content changes in the secrets will not be effective in the apicast deployment.
• The controller implements a custom mapper. The map will:
  • List of APIcast CRs with a label including the ID of the secret generating the event. The search will be scoped to the watched namespaces list.
  • Create one reconcile request for each element in the returned list
• Notify secret content update to the deployments using that secret. This is implemented leveraging pod annotations including secrets resourceVersion. When a secret changes, the resourceVersion changes and so does the annotations of the apicast pod. When the annotations of the pod (in the deployment pod template) changes, the deployment controller automatically does rollout of the pods to have the new annotations. The new pods will read the updated content of the secrets.

This PR includes:

• [x] The implementation of the smart secret management
• [x] Upgrade procedure to remove ownership of the secrets

Manually tested with the following APIcast CR:

---
apiVersion: apps.3scale.net/v1alpha1
kind: APIcast
metadata:
  name: apicast1
spec:
  resources: {}
  logLevel: debug
  deploymentEnvironment: staging
  configurationLoadMode: lazy
  cacheConfigurationSeconds: 0
  adminPortalCredentialsRef:
    name: supertest
  embeddedConfigurationSecretRef:
    name: apicast1conf
  customEnvironments:
    - secretRef:
        name: env1
  customPolicies:
    - name: Example
      version: "0.1"
      secretRef:
        name: cp-1
  httpsCertificateSecretRef:
    name: mycertsecret
  openTracing:
    enabled: false  # if activated, apicast requires opentracing agent up and running
    tracingConfigSecretRef:
      name: tracingconf

• [x] apicast CR includes labels with secret UIDs

  apiVersion: apps.3scale.net/v1alpha1
  kind: APIcast
  metadata:
  annotations:
  apicast.apps.3scale.net/operator-version: 0.6.0
  generation: 2
  labels:
  secret.apicast.apps.3scale.net/52d17bc0-d0ce-4779-91cf-7cd7ab057751: "true"
  secret.apicast.apps.3scale.net/77b3d529-b6cc-4d78-a4aa-c1eb6d7426cb: "true"
  secret.apicast.apps.3scale.net/5670d380-cee8-42af-8588-57cfcf61aa74: "true"
  secret.apicast.apps.3scale.net/06659bae-b666-44b7-a396-b81d909bf797: "true"
  secret.apicast.apps.3scale.net/cad7654e-71b3-453f-a779-d1c8ab6140ba: "true"

• [x] deployment pod template annotations with secret resource version

  template:
  metadata:
    annotations:
      apicast.apps.3scale.net/admin-portal-secret-resource-version: "12863311"
      apicast.apps.3scale.net/customenv-secret-resource-version-env1: "12851137"
      apicast.apps.3scale.net/custompolicy-secret-resource-version-cp-1: "12851140"
      apicast.apps.3scale.net/gateway-configuration-secret-resource-version: "12863312"
      apicast.apps.3scale.net/https-cert-secret-resource-version: "12851144"

• Upgrade procedure:
  • [x] admin portal and configuration secret's owner references are removed
  • [x] apicast CR includes labels with secret UIDs
  • [x] pod annotations with secret resource version

[eguzki]

Out of curiosity, typically should changes be manually verified by running the operator as part of a review ?

We have not been doing that, maybe with few exceptions, though. I am not going to tell you not to run it manually. If you do, the better. But not a hard requirement. There is a QE team for e2e tests.