Potential Redesign - Feedback Requested

In this issue I am going to outline some of the major design changes that will (if implemented) break backwards compatibility with the current version of Knife Container. I do not make these decisions lightly but I think that everyone will agree that the changes I am proposing will help move the project further and make it easier to use.

To summarize, here is how the current CLI is broken down:

knife container docker init generates a new Docker context on your file system. You pass in a Docker image name to the CLI as well as various options to specify how you want your image to behave (run_list, base docker image, etc). There are a number of weaknesses in the process.

Syntactically, this does not follow the chef generate pattern that is being established over in the ChefDK (chef generate cookbook, chef generate recipe, etc).
All options must be passed in via the CLI when you run knife container docker init. If you forget a value, or sysadmin-forbid - fat-finger a value, you need to re-enter the entire command. This also leads to a practice that I myself hate: maintaining text documents with nothing but commands for easy copy-pasta.
Options passed in to the CLI are varying levels of difficult to change once they have been specified. If you want to modify your run_list, you need to change the first-boot.json that is generated. If you want to switch between running the chef-client against a server or in local-mode...well good luck.
You must maintain a separate folder for your Docker contexts even if you do not subscribe to the monolithic chef-repo pattern.
To my knowledge, it will be impossible to use the same image across multiple organizations. Also, while Docker is currently the only supported "provider," you will need to maintain a different context for each provider. That means if you want to use a different container provider you will need to maintain the two images separately.

knife container docker build takes in the name of a pre-created context and builds it using Docker. Many of the same weaknesses that I listed above for knife container docker init apply to docker build as well:

Options must be passed through CLI
All contexts must be in the same dockerfiles directory with a very specific naming scheme.
The command itself will only build a Docker image. If you wish to build more than one image, you need to run multiple commands.

Moving forward, I want to make the Knife Container as easy to use for everyone, regardless of their familiarity with underlying technologies like Docker. However, I don't want to restrict the capabilities of the power users to take full advantage of those underlying technologies either. In the subsequent paragraphs I am going to outline changes in the workflow that I believe will make Knife Container even easier to use while still remaining powerful for the hard-core users.

To better follow the workflow being used in the ChefDK, we will start by replacing the knife container docker init command with knife container generate. In addition to making the CLI fit more closely with the chef generate commands of the ChefDK, knife container generate will create Container Image Contexts that are independent of the underlying container provider. You will still have all the freedom to take full advantage of Docker or any other container provider but each managed context will no longer be 1-to-1 with a specific provider.

The knife container generate command will accept a name along with a long list of options. However, specifying the options when you run the command is no longer necessary because the primary vehicle for driving the configuration of the Container Image Context will be the Containerfile that is generated by this command.

By default, when you run knife container generate NAME, a folder with the NAME you provided will be created in your current working directory.

$ pwd
/home/tomduffield/

$ knife container generate my_lamp
...
Context Created: /home/tomduffield/my_lamp

If you want to have a dedicated directory where you keep all of your Container Image Contexts, you can easily set the knife[:containerfiles_path] variable in your knife.rb. If you are using the monolithic chef-repo pattern, you can easily set knife[:containerfiles_path] = "#{chef_repo_path}/containerfiles". The knife container generate command will then create the Container Image Context in the containerfiles directory of your chef-repo.

$ pwd
/home/tomduffield/

$ knife container generate my_lamp
...
Context Created: /home/tomduffield/chef-repo/containerfiles/my_lamp

Much like a Berksfile or .kitchen.yml, the Containerfile identifies that directory as a Container Image Context. The Containerfile itself is ruby file that will be parsed by the knife container commands. In my initial design, it will look like this:

name 'my_lamp'
run_list 'role[base]', 'apache2', 'mysql', 'php'

image 'org1/my_lamp' do
  provider    :docker
  base        'chef/ubuntu-14.04:latest'
  tags        'tag1', 'tag2'
  dockerfile  <<-STEPS
    EXPOSE 80
  STEPS
end

I won't go too heavily into the actual documentation of each of the configuration values inside the Containerfile now because the great thing about the Containerfile is that the options are nearly limitless. My goal is to have an experience that is similar to writing a Chef recipe or working with Chef's Policyfiles.

Next, to build the image(s) you will use the knife container build command. If you do not specify a name, this command will scan the parent directory tree looking for a Containerfile much the same way the knife command looks for the .chef/knife.rb file. If it finds a Containerfile, it will build the associated container images. If it can't find a Containerfile, the command will fail.

If you specify a name (knife container build my_lamp) it will scan the directory tree the same way but in addition to checking for the presence of a Containerfile it will check to make sure the name value in the Containerfile matches the name you specify.

If you specify a name and have set the knife[:containerfiles_path] variable, knife container build will look for a Containerfile in the current working directory. If it can't find one or the names do not match, it will scan the knife[:containerfiles_path] directory for a context with a Containerfile that matches the name you provided. At this point, so we dont end up scanning your entire hard disk, when searching the knife[:containerfiles_path] directory it will only search one directory level deep for a container file.

would it make sense with the separation to rename the command from 'container' to 'image' this is a semantic change, but it would make sense that you could eventually generate AMI/qcow etc images from this. Or at least that seems like a likely direction.

My second thought was around how this relates to whats going on in chef-metal land, and is there potential for a common driver layer between tools ?

And the 3rd thought would be. Aren't you starting to look alot like packer at this point ? Why not just make some stuff around that existing tool that could make it easy to use a repo/runlist/role in a packer build ?

Before I write my own ideas, let's provide some info for @spheromak (and others reading this later) which I also investigated:

I asked around on the Chef mailing list regarding the correlation between chef-container and chef-metal. You can read the mail thread here.
Second: to give you an idea what you get with Chef and the container support is @tduffield talk titled Managing Containers with Chef. In short: knife-container builds your container, with chef-init configured as your PID1 process.
After we have our containers created, how do we run multiple of them in a scalable setup? This is where chef-metal comes into the picture, and nothing better than a good demo by Mandi Walls. Again, in short: chef-metal is a Chef ruby library offering you machine and machine_image resources and providers so you can idempotently define your application setup, datacenter setup, .... While not specifically tied to Docker, when using it with the Docker containers created in part 2, you can have your containers run Chef against an external Chef server (zero or real).

Regarding your questions, I can agree that there is overlap with Packer, but the Chef container support solves the PID1 problem where Packer doesn't. Chef (the community) are not the only one to tackle this, but I like the way things are progressing now. Not only can I use a tool for my config management on a single node, I can use the same tool for my complete cluster, application setup or even datacenter. I wondered if we didn't end up using Chef as that hammer where all our problems looked like nails, but in this case it fits the bill nicely (IMO). Packer can use Chef to provision the containers it creates, but it also hasn't solved the fact that everything has to be passed in on the command line. See the last section of the Packer Docker builder page.

Hope this helps in understanding how everything fits together.

Now on to my comments for this proposal.

I applaud the fact that the command line options would move to a Containerfile. What's uncertain from your description is if I could put a single container context in a separate Git repo. Why? My work with Chef fell into place when Berkshelf came up and I could start with the single-repo-per-cookbook pattern. I hope that your proposed changes would allow me to create a single-repo-per-container-context. This would allow me for a Git repo containing my application code, a repo containing the cookbook to deploy my code and a repo that ties both together to build the container.

Do you see this happen? Would such a container context contain or allow for provider specific files?

In your presentation (which I mentioned earlier), you talked about the difference of the volume layers when using plain Docker or using Chef with it. Chef has an advantage indeed, but how does this work in real life? My official Docker containers would be built by CI agents. You may not assume that subsequent runs of knife container build can build on top of the local state of a previous run. This means that every run will need to tag and push its results (something Packer does with a post-processor). The next run will need to have it's FROM changed to the last tag if you want to create a volume with only the changes. Will you have support in knife-container to bump this "version" number? Or would this rather fit in stove?

I don't know if knife-container is the place for this, but how will Docker data persistency be supported? Managing data in containers mentions to create containers for portable storage, from where you can backup and/or restore. Do you have any idea where this fits in the plan? Or is this rather something for chef-metal?

@ringods thanks for the additional info and your feedback. To answer your questions:

My goal would be to allow you to put the containerfile whether you please. If you want to put them all into a dedicated directory, awesome. If you want a one-repo-per-containerfile, also awesome.

You bring up an interesting point regarding the automated building and the layers. I think I know what the solution for that would look like but I want to put some more thought into it before I put it out onto the internet where it will live forever.

Managing your data persistency is not a use case that I had in mind for knife-container. It would allow you to generate an image that would persist data but that is where its responsibilities would end. I think this would realistically be an integration with chef-metal if it were to fall into the purview of Chef at all.

@ringods thanks for the links to info. All of this I was aware of before commenting. I still think my original questions deserve more direct response. You could use packer here and still use chef-init. Never mind the discussion/issues around putting a ruby process as pid1. Metal could do the container building for you. I am trying to clarify where knife-container fits in the future. If we already have things that do alot of whats going on in container we can focus on making those things better vs making a new tool ? or possibly something more generic for the community to use.

To be clear I am not a naysayer around knife-container these are just genuine questions I have regarding where metal and container converge.

@ranjib showed this stuff to me at summit maybe he showed it to you too @tduffield ? https://github.com/GoatOS/base/blob/master/cookbooks/goatos/files/default/goatos-meta.rb#L87-L114 which is a chef-converge in a container without the chef-client being present. Something like this IMO is more how I would want to run my containers in prod (vs having to have a full omni-chef install + chef-init). Bringing all this up since you're considering redesign.

@tduffield re: layers. What would be AMAZING would be a resource mutation to commit a layer in docker. (never mind getting to the point where you could restart a converge at that point). but this would be pretty slick if it could be pulled off. I suspect some lower level docker-chef fu in order to pull it off tho.

this is more relevant code regarding chef bases container building

I would would be nice if this project is renamed to knife-docker or something else, since its not really a container abstraction. Check this for technical details on linux containers. What @ringods descibe as PID 1 issue, is the actual strength of docker or application containers (philosophically they are more like servlet containers), which is managed by a container management system (docker daemon), other linux containers dont need this. They have init inside them. Hence chef-init is a specific requirement for docker. Other container implementation, like openvz, solaris or even LXC wont need this. In contrast if you force an application container to behave like a linux container it might interfere with downstream tooling (like fleet etc). I also think that if you still take this route rewrite chef-init in C or some other language, it will be sub optimal to supervise a db or jvm based app to be supervised by a ruby process.

Hey @ranjib - thanks for your feedback.

I think that this project (knife-container) is named correctly. While it is currently only tailored for Docker the goal is that in the future it will support other container models as well, including LXC. I would love feedback from you on what we could do to allow knife-container to model other container workflows besides Docker.

You are correct in that chef-init solves a Docker-specific problem, specifically the lack of an init. For users that don't want or need chef-init, there is no actual requirement that users leverage it if they are using chef-container or knife-container. If that is not clear, that is a gap in the documentation.

As for chef-init being in Ruby, the process that does the actual supervision is runit. The actual chef-init process pretty much parses input and spawns runit. I am not against rewriting chef-init in another (compiled) language but it just hasn't been high on my priority list. If others feel very strongly about it as well I will increase its priority.

@spheromak I agree that both a) being able to converge a node without chef being present on the node and b) having a chef resource that can trigger a container image layer commit are fantastic features. Those have been on my roadmap for awhile and your feedback calling for them helps me prioritize them.

So this "knife container" re-design, as you describe, will change the 'init' and 'build' steps mechanism, but not going to change how chef-init runs etc....Is that correct?

I'd also suggest trying to support Docker host being separate from Knife Chef workstation.

chef-boneyard / knife-container

Potential Redesign - Feedback Requested #41