Repository organisation for Flux?

[mikecann]

Hi,

I am trouble understanding how flux is supposed to detect if there is a new version of an image.

We have a service called "fraas" that is made up of two containers "app" and "web". "app" is the app code and "web" is nginx webserver.

Suppose a developer makes a change to some code then pushes. This kicks off the CI pipeline to build the docker images they are pushed to:

{CUSTOMERID}.dkr.ecr.us-east-1.amazonaws.com/fraas:app

and

{CUSTOMERID}.dkr.ecr.us-east-1.amazonaws.com/fraas:web

So every time we make a change it builds new images and uses the same tags "app" and "web" is this correct?

Im guessing its not as it doesnt seem to auto-update the cluster for us.

How does flux know how to update the cluster?

I noticed that your flux helloworld example is using the image like:

quay.io/weaveworks/helloworld:master-a000001

and

quay.io/weaveworks/sidecar:master-a000001

So does that mean you have a separate image repository per sub-container then you tag based on the branch (e.g. "master") and the commit (e.g "a000001") ?

In which case our images should look like:

{CUSTOMERID}.dkr.ecr.us-east-1.amazonaws.com/fraas-app:master-a000001

and

{CUSTOMERID}.dkr.ecr.us-east-1.amazonaws.com/fraas-web:master-a000001

Is this correct? It shouldnt be too hard to have our CI do this but then im confused as to how flux knows which version to update the cluster with if the tag is continually changing?

Sorry for such a long-winded way of asking this question, I hope it makes sense I really appreciate all your hard work on flux and weaveworks cloud.

[squaremo]

Hi @mikecann,

Your latter intuition is correct: the idea is to give each distinct image version its own tag. For our own use of flux, we use the branch and shortened git SHA1.

Flux looks at the image layer creation time to determine the ordering of images and selecting which is the most recent.

This is almost always fine but there are a couple of improvements that we will at some point make:

• looking at a build time label to determine the ordering rather than using the layer creation time; this is because in some circumstances Docker will reuse an old image layer (if the contents are the same), and this upsets our ordering.
• other modes of ordering and selecting the latest image, including using semver.
• supporting image digests, which exactly identify an image, instead of tags

HTH!

[mikecann]

Great, thanks for getting back to me.

So whats the best practices on managing different environments?

Our git repo "fraas" has two branches "master" which we use for dev and "prod".

Our CI system would then be configured to build from both branches so for example we would now have the following images:

{CUSTOMERID}.dkr.ecr.us-east-1.amazonaws.com/fraas-app:master-a000001 {CUSTOMERID}.dkr.ecr.us-east-1.amazonaws.com/fraas-web:master-a000001

{CUSTOMERID}.dkr.ecr.us-east-1.amazonaws.com/fraas-app:prod-b000002 {CUSTOMERID}.dkr.ecr.us-east-1.amazonaws.com/fraas-web:prod-b000002

Then I guess we would have two clusters, one for dev and one for prod.

But how then would Flux know to only use "prod" tags in the "prod" cluster?

Or does that mean we have to have separate image repositories for each environment?

{CUSTOMERID}.dkr.ecr.us-east-1.amazonaws.com/fraas-app-prod:prod-b000002 {CUSTOMERID}.dkr.ecr.us-east-1.amazonaws.com/fraas-web-prod:prod-b000002

[squaremo]

This is the bit where lots of different approaches can work ..

You can filter the tags that are considered for automation, so you could have for example:

• Dev gets fraas-web:dev-*
• Prod gets fraas-web:prod-*

This is not super-well documented, but you can check fluxctl help policy for some details on how to set a filter. You would still have to have a script somewhere (e.g., CI) that's promoting the images by retagging them.

In the Weave Cloud infrastructure we use exactly the same set of image tags in dev and prod, and promote image versions manually. Each of the clusters has its own directory in the git repo (they might equivalently just have different git repos -- it would mean duplicating a bunch of stuff, but logically it wouldn't matter). One reason to prefer this is that it's easier to track problems with an image, since it's named the same everywhere.

[mikecann]

Oh cool, thanks for those comments.

So I spent today experimenting a little more and cant seem to get flux to auto-update even though I think I have everything correctly configured.

So I started with a clean minkube and flux install.

I have a kubernetes microservice config in a git repo called "FluxExperiments" with the following config:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    flux.weave.works/automated: "true"
    flux.weave.works/tag.app: glob:app-kubernetes-*
    flux.weave.works/tag.web: glob:web-kubernetes-*
  name: fraas-deployment
spec:

  replicas: 1

  selector:
    matchLabels:
      app: fraas

  template:
    metadata:
      labels:
        app: fraas

    spec:

      containers:

        # This contains our laravel app, and the php runtime
        - name: app
          image: 080666409137.dkr.ecr.us-east-1.amazonaws.com/fraas:app-kubernetes-36656675664820a24c5c4bce69d6dc6da999407e
          ports:
            - containerPort: 9000

          # Our secret environment variables
          envFrom:
            - configMapRef:
                name: fraas.config          

        # Web is the nginx webserver
        - name: web
          image: 080666409137.dkr.ecr.us-east-1.amazonaws.com/fraas:web-kubernetes-36656675664820a24c5c4bce69d6dc6da999407e
          ports:
            - containerPort: 80

      # To pull the docker images from a private repo
      # we need to supply AWS login credentials
      imagePullSecrets:
        - name: awsecr-cred
---
apiVersion: v1
kind: Service
metadata:
  name: fraas
spec:
  selector:
    app: fraas

  # Because we are running in minikube for now we 
  # dont have a load balancer and are just using nodeport
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http

I also have a "gvweb" service but it looks pretty much the same as above.

When I deployed flux with it pointing to the repo that contained the above config we successfully had our cluster up and running, so far so good.

Now I made a change to the src for "fraas" and pushed it. The CI then built and pushed the image.

I waited some time expecting flux to now update but unfortunately it doesnt seem to have updated.

When I type fluxctl list-images I get the following:

± % fluxctl list-images                                                                                                                                                                                                           !1277
CONTROLLER                           CONTAINER  IMAGE                                                        CREATED
default:deployment/flux              flux       quay.io/weaveworks/flux                                      
                                                |   master-3d56be4                                           23 Mar 18 16:00 UTC
                                                |   master-7f1b0fa                                           23 Mar 18 11:34 UTC
                                                |   master-0e970a1                                           22 Mar 18 15:14 UTC
                                                |   master-f220230                                           20 Mar 18 15:59 UTC
                                                |   master-866da34                                           20 Mar 18 00:52 UTC
                                                |   master-9da7c57                                           19 Mar 18 16:43 UTC
                                                '-> 1.2.5                                                    19 Mar 18 14:58 UTC
                                                    master-a3149f3                                           19 Mar 18 13:58 UTC
                                                    master-6576049                                           19 Mar 18 11:00 UTC
                                                    master-6492293                                           16 Mar 18 15:02 UTC
default:deployment/fraas-deployment  app        080666409137.dkr.ecr.us-east-1.amazonaws.com/fraas           
                                                |   web-kubernetes-effdbbaa4d5b9f14c0b8a6cf85799ab0b1205b58  28 Mar 18 04:07 UTC
                                                |   app-kubernetes-effdbbaa4d5b9f14c0b8a6cf85799ab0b1205b58  28 Mar 18 04:07 UTC
                                                |   web-kubernetes-36656675664820a24c5c4bce69d6dc6da999407e  28 Mar 18 03:44 UTC
                                                '-> app-kubernetes-36656675664820a24c5c4bce69d6dc6da999407e  28 Mar 18 03:44 UTC
                                                    web-kubernetes-d918e3865bd74516393a1ede8b26de67c8e0cc4d  28 Mar 18 03:07 UTC
                                                    app-kubernetes-d918e3865bd74516393a1ede8b26de67c8e0cc4d  28 Mar 18 03:07 UTC
                                                    web                                                      27 Mar 18 07:34 UTC
                                                    app                                                      27 Mar 18 07:34 UTC
                                     web        080666409137.dkr.ecr.us-east-1.amazonaws.com/fraas           
                                                |   web-kubernetes-effdbbaa4d5b9f14c0b8a6cf85799ab0b1205b58  28 Mar 18 04:07 UTC
                                                |   app-kubernetes-effdbbaa4d5b9f14c0b8a6cf85799ab0b1205b58  28 Mar 18 04:07 UTC
                                                '-> web-kubernetes-36656675664820a24c5c4bce69d6dc6da999407e  28 Mar 18 03:44 UTC
                                                    app-kubernetes-36656675664820a24c5c4bce69d6dc6da999407e  28 Mar 18 03:44 UTC
                                                    web-kubernetes-d918e3865bd74516393a1ede8b26de67c8e0cc4d  28 Mar 18 03:07 UTC
                                                    app-kubernetes-d918e3865bd74516393a1ede8b26de67c8e0cc4d  28 Mar 18 03:07 UTC
                                                    web                                                      27 Mar 18 07:34 UTC
                                                    app                                                      27 Mar 18 07:34 UTC
default:deployment/gvweb-deployment  app        080666409137.dkr.ecr.us-east-1.amazonaws.com/gvweb           
                                                |   web-kubernetes-45b3db01dd0713341a9f4ec6c70890ad51a296bb  28 Mar 18 03:06 UTC
                                                '-> app-kubernetes-45b3db01dd0713341a9f4ec6c70890ad51a296bb  28 Mar 18 03:06 UTC
                                                    web                                                      26 Mar 18 09:00 UTC
                                                    app                                                      26 Mar 18 09:00 UTC
                                     web        080666409137.dkr.ecr.us-east-1.amazonaws.com/gvweb           
                                                '-> web-kubernetes-45b3db01dd0713341a9f4ec6c70890ad51a296bb  28 Mar 18 03:06 UTC
                                                    app-kubernetes-45b3db01dd0713341a9f4ec6c70890ad51a296bb  28 Mar 18 03:06 UTC
                                                    web                                                      26 Mar 18 09:00 UTC
                                                    app                                                      26 Mar 18 09:00 UTC
default:deployment/memcached         memcached  memcached                                                    
                                                |   1                                                        14 Mar 18 06:26 UTC
                                                |   1.5                                                      14 Mar 18 06:26 UTC
                                                |   1.5.6                                                    14 Mar 18 06:26 UTC
                                                |   latest                                                   14 Mar 18 06:26 UTC
                                                |   1-alpine                                                 03 Mar 18 01:19 UTC
                                                |   1.5-alpine                                               03 Mar 18 01:19 UTC
                                                |   1.5.6-alpine                                             03 Mar 18 01:19 UTC
                                                |   alpine                                                   03 Mar 18 01:19 UTC
                                                |   1.5.5                                                    17 Feb 18 08:37 UTC
                                                |   1.5.5-alpine                                             13 Feb 18 22:48 UTC
                                                : (39 image(s) omitted)                                      
                                                '-> 1.4.25                                                   24 May 16 03:02 UTC

So it looks like Flux successfully recognises that there are new images (effdbbaa4d5b9f14c0b8a6cf85799ab0b1205b58) but for some reason doesnt auto-update the cluster and im not sure why.

The only src change that was made to fraas is a one-line change of some HTML just so I can see a change on the screen. So I dont think its anything wrong with the container.

This is a few of the tail lines from the flux log:

ts=2018-03-28T05:14:43.282491733Z caller=images.go:18 component=sync-loop msg="polling images"
ts=2018-03-28T05:14:47.287226045Z caller=images.go:57 component=sync-loop service=default:deployment/gvweb-deployment container=app currentimage={AWSID}.dkr.ecr.us-east-1.amazonaws.com/gvweb:app-kubernetes-45b3db01dd0713341a9f4ec6c70890ad51a296bb repo=080666409137.dkr.ecr.us-east-1.amazonaws.com/gvweb pattern=app-kubernetes-*
ts=2018-03-28T05:14:47.287302442Z caller=images.go:57 component=sync-loop service=default:deployment/gvweb-deployment container=web currentimage=080666409137.dkr.ecr.us-east-1.amazonaws.com/gvweb:web-kubernetes-45b3db01dd0713341a9f4ec6c70890ad51a296bb repo=080666409137.dkr.ecr.us-east-1.amazonaws.com/gvweb pattern=web-kubernetes-*
ts=2018-03-28T05:14:47.287328667Z caller=images.go:57 component=sync-loop service=default:deployment/fraas-deployment container=app currentimage=080666409137.dkr.ecr.us-east-1.amazonaws.com/fraas:app-kubernetes-36656675664820a24c5c4bce69d6dc6da999407e repo=080666409137.dkr.ecr.us-east-1.amazonaws.com/fraas pattern=app-kubernetes-*
ts=2018-03-28T05:14:47.287346637Z caller=images.go:62 component=sync-loop service=default:deployment/fraas-deployment container=app currentimage=080666409137.dkr.ecr.us-east-1.amazonaws.com/fraas:app-kubernetes-36656675664820a24c5c4bce69d6dc6da999407e msg="added image to changes" newimage=080666409137.dkr.ecr.us-east-1.amazonaws.com/fraas:app-kubernetes-effdbbaa4d5b9f14c0b8a6cf85799ab0b1205b58
ts=2018-03-28T05:14:47.287664824Z caller=images.go:57 component=sync-loop service=default:deployment/fraas-deployment container=web currentimage=080666409137.dkr.ecr.us-east-1.amazonaws.com/fraas:web-kubernetes-36656675664820a24c5c4bce69d6dc6da999407e repo=080666409137.dkr.ecr.us-east-1.amazonaws.com/fraas pattern=web-kubernetes-*
ts=2018-03-28T05:14:47.287710494Z caller=images.go:62 component=sync-loop service=default:deployment/fraas-deployment container=web currentimage=080666409137.dkr.ecr.us-east-1.amazonaws.com/fraas:web-kubernetes-36656675664820a24c5c4bce69d6dc6da999407e msg="added image to changes" newimage=080666409137.dkr.ecr.us-east-1.amazonaws.com/fraas:web-kubernetes-effdbbaa4d5b9f14c0b8a6cf85799ab0b1205b58
ts=2018-03-28T05:14:47.291973152Z caller=loop.go:102 component=sync-loop event=refreshed url=git@github.com:GenvisAU/FluxExperiments.git branch=master HEAD=9051480013109235dac1c7da19ba3a679a228fc5
ts=2018-03-28T05:14:47.29210873Z caller=loop.go:110 component=sync-loop jobID=eebc6a8e-54b6-7580-b1cf-483c563d4e66 state=in-progress
ts=2018-03-28T05:14:47.32092979Z caller=loop.go:120 component=sync-loop jobID=eebc6a8e-54b6-7580-b1cf-483c563d4e66 state=done success=false err="did not update expected containers: web"
ts=2018-03-28T05:15:41.9365045Z caller=release.go:61 component=cluster method=Sync cmd=apply args="--namespace default" count=4
ts=2018-03-28T05:15:42.93854736Z caller=release.go:100 component=cluster method=Sync cmd="kubectl --namespace default apply -f -" took=1.001850513s err=null output="service \"fraas\" unchanged\ndeployment \"fraas-deployment\" unchanged\ndeployment \"gvweb-deployment\" unchanged\nservice \"gvweb\" unchanged"
ts=2018-03-28T05:15:42.938616604Z caller=release.go:61 component=cluster method=Sync cmd=apply args= count=2
ts=2018-03-28T05:15:43.154901647Z caller=release.go:100 component=cluster method=Sync cmd="kubectl apply -f -" took=216.240087ms err=null output="configmap \"gvweb.config\" unchanged\nconfigmap \"fraas.config\" unchanged"

I think the interesting line in there is:

ts=2018-03-28T05:14:47.32092979Z caller=loop.go:120 component=sync-loop jobID=eebc6a8e-54b6-7580-b1cf-483c563d4e66 state=done success=false err="did not update expected containers: web"

Im not sure what did not update expected containers: web" means?

[mikecann]

BTW, I would be happy to update the docs on tagging (https://github.com/weaveworks/flux/issues/912) once I understand how its supposed to work and ensure im not doing anything wrong

[squaremo]

@mikecann There are a few caveats around the manifests that flux can update -- see https://github.com/weaveworks/flux/blob/master/site/requirements.md. In your case, I think the thing that preventing the updates is having the deployment and service in the same file. Simply putting the deployment in a separate file should be enough for flux to operate on it. Oh -- and outdent the containers one level, i.e., instead of

containers:
  - name: foo
    image: bar/v1

make it like

containers:
- name: foo
  image: bar/v1

Both are valid YAML, but for horrible regex-related reasons, flux is sensitive to the indentation of those specific blocks.

(Work is underway in #976 on removing some of these limitations, which arise because we try to preserve comments and whitespace.)

[mikecann]

@squaremo Oh nice! Yes that fixed it, it now works as expected :+1:

I had read that requirements before but I guess I must have missed that point about them being in sepparate files.

Thank you very much for all your help.