Design Doc: Support custom annotations on Deployments

[Cryptophobia]

Problem

"I cannot add custom annotations to my Hephy app's Deployments or Pods because they will be overwritten and lost when I scale or deploy a new release." :crying_cat_face:

Goal

Allow users to set custom annotations on Hephy Deployments manually and have the controller preserve those annotations to future releases.

This hack will allow users to be able to use custom annotations on Pods through modifying the Deployment object. Which is useful for using things like kube2iam, istio.io, kubedb and many others.

Modify (hack) the controller so that Deployment object rolls forward the custom annotations into future releases. This will require only a change in the controller.

Code Changes

• Add a dictionary of the deployment object annotations and check for existence of this hash map in the scheduler init functions. If the annotations dict is present, set it forward into the new deployment manifest function. If the annotations dict does not exist, do not set anything. Always initialize a blank annotations dict through **kwargs on manifest function or other functions that depend on the functionality of the manifest function.

Tests

Write some tests on the controller to verify that annotations can be set on Deployments and that they exist on the Pods created by the Deployments. :+1:

[Cryptophobia]

Just for reviews, this can probably be merged today. I will link the PR here and close this through the PR when done.

Made Hephy support custom annotations on deployments so the annotations are set on the deployment's child pods. Requires manual edit on the deployment:

kubectl edit deployment rocksolidapp-web -n rocksolidapp

Then edit the part:

...
spec:
  ...(skipping this part)
  template:
    metadata:
      annotations:
        anton: antonwuzhere
        iam.amazonaws.com/role: role-arn-pods
        nginx.ingress.kubernetes.io/app-root: /rootfs
        sidecar.istio.io/inject: "true"
        testing: testing
      creationTimestamp: null
      labels:
        app: rocksolidapp
        heritage: deis
        type: web
        version: v39
      name: rocksolidapp-web
      namespace: rocksolidapp

and add the annotations on the deployment['spec']['template']['metadata']['annotations'] and Hephy Workflow v2.20.0 preserves these annotations for all releases after that. :thumbsup:

This will add quick support for kube2iam annotations on pods now since you can add them to each deployment object managed by Hephy and they will not be overwritten when a new deployment happens or a new release is created (scale events and others).