Source docker images (internal)

[mcasperson]

See this blog post (internal only).

We want your feedback to determine if this feature is a good fit for Octopus. Specifically we would like to know:

• Does the idea of environment specific images make sense?
• Does the ability to generate container images on the fly make sense?
• Will these proposed features solve problems you have seen customers encountering?
• Would the concept of Source Docker Images be valuable for your teams in terms of sales, marketing, or solutions?

[ryangribble]

I really like this concept and think that although it does "violate" the pure immutability concept of containers, it still aligns with the basic concept of CI server builds an immutable artifact -> Octopus deploys and promotes that artifact through a life cycle of environments, handling the per environment configuration in a central and seamless way.

The fact that under the covers we are helping to transparently rebuild images with injected config is 💎 and it brings all the power of structured config file variable replacement etc that is a strong suit of Octopus and an expectation of our customers, to a modern/containerised eco system. I think its the best of both worlds!

I like using deployment specific tag for the image versions as well, although you could consider whether the environment ID could be included as well to be a helpful visual cue (when looking at a list of container versions i could at least see which ones were all for the same environment rather than a deployment ID on its own i have no context of which ones were for the same environment or not) 🤔

Other feedback would be that the document title or main emphasied concept seems to be the "Source Docker Image" whereas to me that is just the underlying tech mechanism powering this... its the CAPABILITY and POTENTIAL that I like here, eg being able to use the power of Octopus to deliver environment based configuration on (effectively) immutable container images... so I would definitely say we should focus on that aspect and potentially not just for external marketing but even when internally talking about the feature/concept. Focusing more on what it unlocks rather than how it's going to work...

[MJRichardson]

Out of curiosity, assuming a pretty typical scenario of wanting to transform a config file in an image, what would the dockerfile roughly look like for that?

[mcasperson]

Out of curiosity, assuming a pretty typical scenario of wanting to transform a config file in an image, what would the dockerfile roughly look like for that?

I think that could go two ways.

The easiest way would be that the application binaries or code would be uploaded as a package. That package is listed as a package reference (webapp in this example) on the SDI, and any configuration files would be selected for processing before the Docker image is built:

FROM node:10
WORKDIR /usr/src/app
COPY webapp .
EXPOSE 5555
CMD [ "node", "index.js" ]

If you already had a "golden" image that you wanted to customise (as opposed to building from scratch), my understanding is the top level Dockerfile looks like this:

FROM genericimage
COPY genericimage/usr/src/app/settings.json /usr/src/app/settings.json

The new image inherits all the settings from the base image (ports, volumes, environment vars etc), so you just add one file and you are good.

The catch here is when the config file to be transformed comes from the image itself. We could look at allowing Docker images to be downloaded and extracted like any other package (tools like umoci unpack do this today). In which case the genericimage directory comes from the genericimage image extracted by Octopus like any other package.

[mcasperson]

Actually we could streamline the use case of building a new image after updating some of it's own files. The Dockerfile would just be something like this:

FROM genericimage
#{for file in all_the_processed_files}
COPY genericimage/#{file} #{file}
#{/endfor}

You select the base image and list the files to be processed, and we do the rest.

[BobJWalker]

I'm thinking about what this looks like at scale.

• Depending on the dockerfile you are looking at 1 minute to 30 minutes getting added to the deployment.
• Can you have multiple dockerfiles per deployment process?
• How long will the docker images exist on the worker? Will they ever get cleaned up?
• How can I share a docker file between projects / spaces? Windows docker images are massssive

[mcasperson]

I have a worker pool with 10 workers. Each worker would need to build that image. Depending on the dockerfile you are looking at 1 minute to 30 minutes getting added to the deployment.

I think we'd have a step that took an SDI, built it, pushed it, and exposed the resulting image as a variable for any subsequent steps to use. So you'd only wear the cost of a build once, and then the subsequent workers would simply pull the image like normal.

Assuming you have a dedicated worker for building that initial image, and assuming the packages don't change, Octopus would complete the step quickly because all packages are cached, and Docker would complete the step quickly because all layers are cached.

Can you have multiple dockerfiles per deployment process?

Each SDI can have it's own Dockerfile, and a deployment process can use as many SDIs as needed.

How long will the docker images exist on the worker? Will they ever get cleaned up?

We didn't really get into the detail of this. I think the answer would be the same for container images today, which I suspect is they never get cleaned up, which is probably not ideal. This is something to look into.

How can I share a docker file between projects / spaces? Windows docker images are massssive

I think this is two questions.

Sharing a Dockerfile (as in the literal file) would be done by sharing a package with the Dockerfile. SDIs could then reference the same package to read the same Dockerfile.

To share the same resulting Docker image, we'd rely on Docker caching. If all your SDIs were based on the same Windows base image, Docker will pull that once and then apply the SDI specific changes on top, which should be minimal.

Docker caching would also allow you to build a shared based image. For example, if all your images use Windows with some other large tooling on top (maybe you need to install MsSQL cli tooling for example), you would build and push that common base image outside of Octopus and base the SDIs on it.

In summary, I think at scale we rely on the fact that pushing Octopus packages is cheap when they don't change thanks to delta copies, and building and pushing a Docker image is cheap when nothing changes because Docker is smart enough to reuse existing cached layers. As long as workers are long lived, they will retain Docker layers and downloading/building images will optimized for us.

Does that address the concerns you had @BobJWalker ? Or I can go into some more detail if you like.

[pawelpabich]

How resource intensive is the process of building docker images? We might need to provision customers with larger Dynamic Worker VMs.

[mcasperson]

You can probably use as much memory, cpu, and disk space as you want when building an image, especially when people could potentially run tests as part of the build.

Given we control the execution of Docker we can set a memory limit as part of the build via the -m option.

Perhaps the answer here is that the code that builds the image is smart enough to catch a failure and check if all the disk or memory was consumed, capture that in a easily reported way, and make a decision about VM sizing if it was a roadblock for enough people?

[pawelpabich]

, capture that in a easily reported way, and make a decision about VM sizing if it was a roadblock for enough people?

We could scale VMs up every time they hit a bottleneck but this not a financially variable option.

It feels like Octopus is becoming a build server which in itself is not a bad thing but we might want to charge for that separately.

[mcasperson]

We could scale VMs up every time they hit a bottleneck but this not a financially variable option.

I'd have to guess the majority of people would be fine with the existing workers, but I think from a product point of view we'd be happy to say that the first milestone of this feature would require customers brought their own workers for workloads that went beyond the capacity of a dynamic worker.

But it is a good scenario to measure, so that would be an important part of the pitch.