search

tailhook / vagga

Vagga is a containerization tool without daemons
http://vagga.readthedocs.org
MIT License
1.86k stars 96 forks source link

Add clean-room mode for building containers #315

Open tailhook opened 8 years ago

tailhook commented 8 years ago

Overview

Unlike using Dockerfile we build containers with mounted /work directory. This means we can use anything from /work to determine what to use to build a container. For example package.json and requirements.txt. Also we have !Depends command which marks files that can influence container hash in case vagga can't determine it automatically.

Problem Statement

There is no way to check whether !Depends and our commands like Copy or NpmDepends (which have implicit dependency on a file) have covered everything we depend on. This is especially valuable for building deployment containers.

Solution

Add vagga _build --clean-room container-name. This flag is going to skip mounting /work. But will create an empty /work folder instead put all the files that build depends on implicitly or explicitly. And run normal build.

I think it's good to be a flag for performance reasons. You need this just to be sure, not for daily use.

Questions

  1. Shouldn't the flag default to true? (maybe in some very future version)
  2. Shouldn't it be a flag on container config, rather than command-line?
  3. Maybe use some smart optimization, like hardlinking files or overlayfs or mountpoints?
  4. Should the flag also check if a build doesn't modify any additional files?
  5. Should this build be temporary, or using this container for running commands is fine too? (like for example container may depend on ls some-dir/* and build successfully and has the same hash, but not same contents as the one build in a normal way)

/cc @popravich, @anti-social

popravich commented 8 years ago

I think it would be better to have it as container config option rather then command line. Also we can make this (and probably auto-clean) option overridable in command line.

popravich commented 8 years ago

So such a container (or a build) is more intended for deploy and not for running processes in it, am I right?

popravich commented 8 years ago

How /work would be mounted if this container is used in command?

tailhook commented 8 years ago

So such a container (or a build) is more intended for deploy and not for running processes in it, am I right?

No. To check the correctness of a build configuration. And yes, not to run commands.

How /work would be mounted if this container is used in command?

As usual. We change only build process.

tailhook commented 8 years ago

I think it would be better to have it as container config option rather then command line.

So which containers deserve the option?

As I've outlined above I would use normal build process for everything including CI (for performance reasons). And run nightly or weekly CI job to check if --clean-room build is still fine.

popravich commented 8 years ago

No. To check the correctness of a build configuration. And yes, not to run commands.

I believe the way to check the correctness of build is to run some process that depends on that data. Otherwise users would end ls-something on that /work dir to check if everything has been copied. Am I missing something?

tailhook commented 8 years ago

I believe the way to check the correctness of build is to run some process that depends on that data. Otherwise users would end ls-something on that /work dir to check if everything has been copied. Am I missing something?

I'm thinking about that too. But:

  1. Built tools usually crash on non-existing files. So if they succeed it may be good enough.
  2. The primary use case for this in our company is: a container with compiled javascripts and CSS. So we are unable to test them properly anyway.
popravich commented 8 years ago

Built tools usually crash on non-existing files. So if they succeed it may be good enough.

There can be lots of side-effect-files also.

The primary use case for this in our company is: a container with compiled javascripts and CSS. So we are unable to test them properly anyway.

What about cases when you're deploying, say, python container and you're !Copy-ing py-files into container (not installing project as package)?

tailhook commented 8 years ago

What about cases when you're deploying, say, python container and you're !Copy-ing py-files into container (not installing project as package)?

Can you describe any case where you have failed to add python's dependency, and how clean-room functionality would help you?

popravich commented 8 years ago

I'm not talking about dependencies, I'm talking about project sources -- with vagga command you run project localy (sources are served/executed from /work) but when you need to deploy you must ship project sources inside container to be able to run it. So the way to know if all sources are copied/ok is to start process. I see that clean-room functionality makes less sense in this case