Enforce keel policy during initial deployment

[severity1]

First of all, I really like keel. :)

Given this deployment.yaml;

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: "floop"
  namespace: operations
  labels:
    name: "floop"
    keel.sh/policy: all
    keel.sh/trigger: poll
    keel.sh/approvals: "3"
    keel.sh/approvalDeadline: "24h"
  annotations:
    keel.sh/pollSchedule: "@every 1m"
spec:
  selector:
    matchLabels:
      app: "floop"
  replicas: 1
  template:
    metadata:
      name: "floop"
      namespace: operations
      labels:
        app: "2048"
        component: frontend
   spec:
      containers:
      - name: "floop"
        image: someacct.dkr.ecr.ap-southeast-2.amazonaws.com/banana/floop:0.0.6
        imagePullPolicy: Always
        ports:
        - containerPort: 80

upon initial apply it will always deploy version/tag 0.0.6 and then it will wait for 1minute before it applies the version/tag enforced by keel policy. 0.0.6 -> 0.0.8.

This scenario momentarily leaves your deployment in a bad state. It becomes far worse if your pollSchedule is longer because this means initial deployment will be in a bad state for a longer period of time.

i propose that initial deployments should apply the version/tag enforced by keel policy immediately without waiting for pollSchedule.

As long as "desired state" can be enforced using keel.sh/policy: i don't think we will need flux style gitops.

Of course i can always just rely on editing version/tag directly on this deployment.yaml file but that defeats the purpose of keel policy.

[rusenask]

Hi, there are good and bad things about this :) Quite often I noticed that when you need to rollback to a previous version or even several previous versions (after finding some specific bug in the application) - you don't want keel to update to the latest available version even if there is one. For this strategy I mostly rely on webhooks or GCR events so I get instant updates when I push, but I don't want it to do anything if I am just deploying older version on purpose. Then, once the issue is fixed, I can just push the newest release and Keel will update it to the latest.

The problem you are describing is the initial deployment and how to keep manifests up-to-date. For that I have created a separate little project - https://about.sunstone.dev/#how-does-it-work :D It's an early version but basically it can check with Docker registry during request what's the latest version and generate a template:

containers:                    
        - image: keelhq/push-workflow-example:{{ .version | latestRegistrySemver "keelhq/push-workflow-example" }}
          imagePullPolicy: Always

I will make it open source as well (and easily self-hosted as it's just a single, stateless container). Would it be more helpful to your situation?

There are some further improvements that I have in mind to keep manifests on github in sync with latest docker registry versions.

[severity1]

Hi @rusenask, thanks for the reply, I will have a look at sunstone, although, does this happen for helm charts as well?

[rusenask]

hmm, not sure what's the best course of action for helm in this case. We would need to somehow update values.yml in the git or chart repo probably.

[severity1]

@rusenask the solution for me that allows me to rollback and lets me to also delete/apply deployments.yml is by using multi image-tagging, ie; pushed imagetag :0.0.18 is tagged as :dev,:tag1,:tag2,:0.0.18-dev with forced policy for matching tag.

basically, if i want to promote an image :0.0.18 to :prod i just tag it as :prod and keel will see this and deploy. it also works for rollbacks because i can just retag :0.0.17 as :prod and that will get deployed. this also solves my issue with the deployments.yml files image not being inline with keel deployments because i just have to tell it to always deploy images tagged as :prod. My deployment pipeline now just promotes images to environment by adding an "environment tag".

although, this poses another challenge with visibility on keelui and notifications, i have outlined them in another issue -> https://github.com/keel-hq/keel/issues/431

atm, i am using a mix of policy depending on which environmnet i deploy to.