vfarcic / docker-flow

Docker Flow: Walkthrough
https://technologyconversations.com/2016/04/18/docker-flow/
MIT License
187 stars 31 forks source link

Strategy to handle Docker images with blue/green deployment #5

Open wilkko opened 8 years ago

wilkko commented 8 years ago

Docker Swarm requires that Docker image for the application/service is pushed to a private registry prior to deployment. If I remember right, local image can be deployed to Swarm but rescheduling to another node will fail...? In blue/green deployment scenario, failed deployment would require also Docker images to be rolled back from registry. Better yet, if the revision of the image would not be at hands of users before deployment is verified. How to handle such failure scenario in terms of Docker images?

vfarcic commented 8 years ago

Docker in general (with or without Swarm), requires image to be stored in some registry (private OSS, private from Docker, Docker Hub, Quay, and so on). The exception is if the image is already in the destination server which is a viable option only while testing some image.

No matter the tools, when you build a new image, you should version it before pushing to the registry. I tend to build images with versions, tag as latest, and push both to the registry. It can be something like:

docker build my-service:1.4.57

docker tag my-service:1.4.57 my-service:latest

docker push my-service:1.4.57

docker push my-service:latest

I also tend to use environment variable inside docker-compose.yml that specifies the version that should be deployed. You can see an example in docker-compose.yml#L5. If BOOKS_MS_VERSION is not set, it defaults to an empty string meaning that the latest version will be deployed. On the other hand, if I want to deploy a specific version, all I have to do is set BOOKS_MS_VERSION to the version (e.g. 1.4.57).

In case of blue-green deployment (at least when done with Docker Flow), there is no need to rollback image from registry. Both versions are available on the server (or the whole Swarm cluster) so the previous version can be easily started if there is a need for rollback. The problem is that starting an image is only the first of the tasks. The proxy needs to be reconfigured as well.

I'm brainstorming while I write... There could be the case when we need to rollback to an older version (for example three versions ago). In such a case, we should store the information about each version. With that data, the "deploy" flow step could use a new argument "version" that would behave in a similar way as "scale". If not specified, it would deploy the latest (as it does now). If it is prefixed with the minus sign (-), it would deploy current version minus the value. If the current version is 5 and the value is -2, it would deploy version 3. The plus sign would to in the opposite direction and, finally, if neither plus nor minus sign is set, it would deploy the specified version. Since most people do not use incremental versions starting from 1, we'd need to store the mapping. If we build versions 0.2, 0.3, 0.7, and 1.0, we'd store the following.

1 - 0.2 2 - 0.3 3 - 0.7 4 - 1.0

The suggestion would be to create three new flow steps build and push, and add version argument. They would behave as follows.

build:

push:

deploy (additional features):

I'm sure that I missed quite a few things that will pop up only during the development. This might be a good start. What do you think? Would this proposal fulfill your needs? Do you think it would be useful?

lincze commented 8 years ago

seems good to me.

something i'm thinking about with blue/green deployment is the docker image fragments garbage coming from frequent deployments on the servers.

do you have any strategy to handle it? (i don't have, at the moment, unless the obvious, a separate houskeeping collect/destroy service)

vfarcic commented 8 years ago

That is also something that can be added to the flow. Removal of unused images in servers as well as from the registry.

There's so much that can be done and so little time...

On Tue, Apr 19, 2016 at 4:25 PM lincze notifications@github.com wrote:

seems good to me.

something i'm thinking about with blue/green deployment is the docker image fragments garbage coming from frequent deployments on the servers.

do you have any strategy to handle it? (i don't have, at the moment, unless the obvious, a separate houskeeping collect/destroy service)

— You are receiving this because you commented.

Reply to this email directly or view it on GitHub https://github.com/vfarcic/docker-flow/issues/5#issuecomment-211943888

Viktor Farcic

Blog: TechnologyConversations.com

Books: The DevOps 2.0 Toolkit (http://www.amazon.com/dp/B01BJ4V66M or https://leanpub.com/the-devops-2-toolkit) Test-Driven Java Development ( http://www.amazon.com/Test-Driven-Java-Development-Viktor-Farcic-ebook/dp/B00YSIM3SC )

wilkko commented 8 years ago

Hmm versioned images could work but that's a new image per build/change set. Staging registry might be needed. We could tag and push image with the latest or master tag only after the deployment is verified.

With good pre-deployment tests the image is verified pretty well at build slave and rarely the image seem to be faulty if the deployment tests fails. Not sure anymore if image rollback feature is actually needed.

Build and push features however seem good fit for Docker flow since docker-compose does not have them and might not get them.