Feature: Multi-Stage Builds

[blakedewey]

This is a feature request and thought experiment to go along with #138.

As of CE 17.05, Docker supports Multi-Stage Builds, where intermediate containers are created and destroyed during the build process. This has the effect of minimizing the final image (by removing all components required for building), but also has some effect on caching, as an intermediate image can be pre-built and called into the final build using a FROM line, potentially even pulling it from a Hub. This could be a great addition to Singularity, especially where large builds can create awkwardly large binary files or compilation times are very long.

Has anyone considered this?

[vsoch]

I think this probably works because an intermediate build is a layer. Arguably you could achieve the same in Singularity by way of cleaning up (rm -rf) build and unnecessary directories. What it sounds like you want is more efficiency in building - eg, given that you are using dependency X five times, you should be able to store that as an "intermediate build" and then dump that layer.

This is an interesting question because it would be nice to have, but it potentially breaks reproducibility in making a single container dependent on multiple other things. The workaround for this would be to simply keep building your containers with Docker, and then convert the final minimal thing into a Singularity image. In order to achieve the same functionality as Docker we would need to have some way to merge / combine containers, which gets messy. How would you propose this to work?

[blakedewey]

I don't think it necessarily breaks reproducibility any more than a linear build path would. You would have multiple dependencies, sure, but that could exist in one Singularity spec file as well. You would also be able to reproducibly build any number of combinations of versions for your dependencies.

Docker actually does it pretty simply by allowing copying from intermediate containers. It actually creates the "build" image completely, then allows access to the file system so that you can copy out files. It's not even as complicated as using layers.

[GodloveD]

You can sortof do the same thing now by starting with a Docker Hub image and using docker as your bootstrap agent in your definition file. I agree that it would be wonderful to also be able to do that with containers on shub or with local containers.

[blakedewey]

Do you have an example of that @GodloveD? That may be a very reasonable workaround, as currently I am just working in docker then importing to Singularity (which is working just fine!). As you said it would just be nice to have it natively supported so that I can develop the two side-by-side.

[GodloveD]

Sure. That's how my lolcow-installer works. It pulls down an existing Docker Hub image and then adds a %runscript. But there is nothing preventing you from adding a %post section too if you are so inclined. Here is a simple TensorFlow example that also does something similar.

[vsoch]

all hail the lolcow!

[blakedewey]

Love the lolcow. I have that running fine with %post there. The thing I was looking to do was take the build artifacts from two different docker images together. Say you have another library that you are using in conjunction with Tensorflow (it really makes sense with >2), then you could build each independently and simply pull and combine. Makes it very easy for sharing complex images, where there may be a whole host of software at specific versions that need to be used in the same image.

[vsoch]

The issue is with combining the two - given everything else is equivalent minus two (isolated) libraries, it would be easy to dump both into the same image. Imagine something in the bootstrap spec that might let you do this like:

Bootstrap: docker
From tensorflow/library1
From tensorflow/library2

But then it would be (more likely) that dumping one on top of the other creates some kind of error if a setting isn't honored, etc. Imagine a very simple case of one config file being shared, and then one dump overrides the original and only one library is configured. I can't imagine how messy it would get given some (but not total) overlap of containers.

Would you be willing and able to test this idea? Eg, take some of these docker intermediate images, get their uri to import, and then try manually building your singularity image with a call to import twice with a new singularity container. I'd be interested to know for how many examples it works, what bugs you hit, and simple things like what happens if the order is reversed. I do think it (could?) be possibly considered to have this kind of functionality - after all there is nothing stopping a user from doing singularity import twice for the same container, it would simply be extended (in some way) to bootstrap.

You could also just do the import twice, but that's probably not well preserved. Maybe some kind of orchestration file to create an image, and import some set of layers would be a reasonable workaround?

[blakedewey]

I hadn't thought of multiple import calls. That may be an interesting idea. But I don't believe it has to be that complex.

See the Docker example here: https://medium.com/travis-on-docker/multi-stage-docker-builds-for-creating-tiny-go-images-e0e1867efe5a

You are explicitly copying what you have built in the previous container to specifically avoid what you are talking about.

[bauerm97]

@blakedewey so I read through that article and I believe we can already achieve that functionality, we just don't provide a pretty wrapper for it (yet?).

So let's consider two definition files, both located in the same location:

//build-env.def
Bootstrap: docker
From: ubuntu

%post
    touch /test.cpp

//build-final.def
Bootstrap: docker
From: ubuntu

%setup
    singularity create -s 568 build-env
    sudo singularity bootstrap build-env build-env.def
    singularity shell -w build-env && mv /test.cpp .
    mv test.cpp ${SINGULARITY_ROOTFS}/

I think that if you attempt to bootstrap build-final now, it will build the build-env container, enter it, copy the target out of the container, and then move it into the new build-final container. Obviously this isn't really pretty, but I believe this accomplishes what you're looking for. If this is the use case you're looking for, we can probably build a wrapper for this (in fact, @vsoch might have already written a very similar thing for singularity-hub to minimize image size?)

Something that might make this look pretty is adding a bootstrap keyword that allows you to access the contents of a provided container (or provided definition file). Something like this is what I have in mind

//build-final.def
Bootstrap: docker
From: ubuntu

using: build-env.img=BUILD-ENV

%post
    cp ${BUILD-ENV}/test.cpp /test.cpp

Then you can access the root of the build-env.img container using the BUILD-ENV environment variable.

[vsoch]

@bauerm97 that is really cool! @gmkurtzer what are your thoughts? I take a similar (but different approach) that doesn't aim to break builds into pieces, but to only build the minimal size needed (with some padding). I basically build into a folder first, calculate the size and add some padding, and then run the bootstrap "for real." I think building into a folder would take away some of the need to mount a singularity image with writable - what about doing something like this:

build-env.def

Bootstrap: docker
From: ubuntu

%post
    touch /test.cpp

build-final.def

Bootstrap: docker
From: ubuntu

%setup
    mkdir estimate
    sudo singularity bootstrap estimate build-env.def
    size=$(du -sh estimate | awk '{print $1}')
    mv estimate/test.cpp ${SINGULARITY_ROOTFS}/ 

The reason I added the bit about size is so that we could (if needed) at some point expand the image being built into given that it's not big enough. But arguably you could skip that and just have:

Bootstrap: docker
From: ubuntu

%setup
    mkdir estimate
    sudo singularity bootstrap estimate build-env.def
    mv estimate/test.cpp ${SINGULARITY_ROOTFS}/ 

so then that would look like in final, and note that I changed "using" to "import", and I switched around the order so it's import: [VARIABLE]:[FOLDER]

build-final.def

Bootstrap: docker
From: ubuntu

import: BUILD_ENV=/path/to/example

%post
    cp ${BUILD-ENV}/test.cpp /test.cpp

This is a cool idea, but when you think about it, it really comes down to maintaining a base of quasi images (folders or images) and then having them defined as variables, and grabbing bits at runtime. This seems like a less elegant solution than just have layers, and it still breaks reproducibility because it creates a huge additional dependency of those things.

But on the other hand, I really like the modularity of it, especially since the final image does have all the requirements. I wonder if we could take a simpler approach that just lets the user import some subset of files / locations from an already existing image? Eg, I could have a folder of "dependency images"

/components
     python
     science
     environment

and then when I write a new image and want to use them, I import what I want to where I want:

Bootstrap: docker
From: ubuntu

%import 
BUILD_ENV /components/environment/base1
PYTHON_MODULES /components/python/anaconda3
RUNTIME /components/environment

%post
    cp ${BUILD-ENV}/test.cpp /test.cpp
    cp ${PYTHON_MODULES}/ /anaconda3
    cp ${RUNTIME}/runscript1 /.singularity.d/runscript    

But again, is this really any different from having a %files section? If we allowed the user to define a variable to a path on the host for different groups of files, and then made those paths available in %post, that would be doing the same.

If we did have import, the statements would need to be kept as metadata, likely under labels (/.singularity.d/labels.json) but that's really not so useful because it's unlikely that someone would ever find it again.

I like this idea, but breaking images into pieces is a dangerous game. It's hard enough to encourage good practices for reproducibility without having multiple things to grab from, and is the end goal really just saving a couple minutes of build time?

If we think through this further, I would want the following:

• that this is somehow fundamentally different / improving upon the current way we can add files to images. Right now it just seems like we are taking the extra step to build first, and then add files from the build.
• assurance that the import components are available for all, and not lost and then the image can't be rebuild.
• a complete record of metadata associated with all imports

@gmkurtzer what are your thoughts? Is there some quasi reasonable approach to this? I like the idea a lot for sure, but it gets me a bit nervous going in this direction.

[GodloveD]

😲 -> WOW! This is really cool stuff! I love the way you two are using and abusing the %setup section! I think there are definitely lots of great potential applications for this. As far as encouraging best practices... meh. I'm always on the side of "let's give people as much power as we can". If that means giving user's enough rope to hang themselves, then so be it. Rope is useful for lots of things beyond suicide. 😸

[gmkurtzer]

I would like to have an internal/native way of bootstrapping/building a container from within another container. That can solve lots of various issues and concerns, but does that answer the main request of this issue?

Going back to the original question... I think we can pretty easily add a %post1, %post2, %post3, and snapshot between them leaving intermediate containers. But... how useful is this? I would suggest actually that it would be better to do this by hand. Bootstrap one container, copy the image, bootstrap against the copy with another bootstrap file (without the "BootStrap: ..." line), and then keep going over and over.

BTW, I do love the idea of "abusing" the %setup to create a temporary container! Haha, never thought of that!

[bauerm97]

@gmkurtzer I think the original question is enunciated clearly in the post that was linked. Essentially, building a temporary container with the intention of copying some build artifacts into the final container. I'm not sure that it necessarily is about intermediate containers per se

[vsoch]

I need to think more about this, because it's venturing into the "orchestration and combination" bit more than just "build a single container " (eg, think of the difference between docker-compose and Dockerfile. Those two things should be distinct I think, meaning that we don't want to make it unclear to the user what a Singularity recipe is for, versus "some other thing."

[vsoch]

The amazing thing here is (at least in my mind) I've now stumbled onto what I think might have been Docker's thought process for bringing in layers in the first place - you have users creating images, and there is some need to cache - what's the best way to do it? By tar guzzing up whatever is the difference from one line to the next, of course! And how do we identify it? By the sha256 sum, of course! If we had some method for a user to say "I am defining this previous image to be X, and from X I will copy this file path into my image here" that is basically similar to letting them package up some subset of a previous build, and then add to the current. But it's arguably worse than the Docker approach, because we don't really have a good way to uniquely identify them, or for a user to nicely do the import (other than manual.)

So - in my mind I think we need to do better than both approaches. Here is a dump of thoughts:

An import is a data container

A data container is really no different than some subset of content to put in a container. Thus, for this import, it's not "pieces of other images" that should be import, but rather data containers. They might have data, or software, both are good. We run into the issue given that the dump includes things like environment variables. We can either put the burden on defining what is needed on the user (and not bring any of the environment in with the data imports) or we can attempt to add to a master environment by having a new file created in env/ for each data folder added. The same would go for the other metadata, it would need to be added to preserve information about each addition. This seems like a good rationale for us having the /env folder the way that we do, given that there aren't a huge number of imports that aren't reasonable. Again, the order of operations for what is defined is dependent on the user. If a particular software has it's own environment variables, it arguably shouldn't conflict too badly.

Enforce modularity with rules for dumping

The riskiest thing about dumping things on top of other things is the potential for screwing something up. It pretty much always happens. We need (somewhat) a re-organization of "where is best practice to put things." Minimally, if we enforce a rule that folders dumped cannot overwrite stuff already there, this would work naturally. For example, let's say I have some software that gets installed to /usr/local (like some weird container software I know...) If I were to just dump, I wouldn't actually overwrite anything, because singularity makes it's own folders within each, eg:

• /usr/local/libexec/singularity
• /usr/local/etc/singularity
• /usr/local/bin/singularity

So it would be good practice to install in this manner. For data / files, we would want something similar, starting at some base folder, and then different "data containers" being added as subdirectories to that, never overwriting another one. Eg:

• /data is in the root of my container
  • onto it I dump a data object intended for /data/sub1 - no overlap, no issue
  • then /data/sub2 - no overlap, no issue

Overall I'd have:

/data
    sub1/
    sub2/

If I then came along and tried to dump /data/sub1? I shouldn't be allowed to do that.

Organization of data containers

Hashes are great for uniquely identifying contents, but the problem I have with them is that they are random/messy combinations of things. I would want a data container / package to be a single modular thing - eg, one python package, or one entire dataset. I need to think more about this part, but if we do allow for the generation of data containers, and arguably we could even do away with letting users decide for software and use package managers / repos and then host them somewhere, we could arguably come up with a nice namespace / organizational schema for modular software things. They would have hashes to verify things, but the "thing" that the user sees and writes to import into the container would be a more human friendly uri.

That's all my travel baggged brain can handle for now, haha. Thoughts?

[blakedewey]

Here is another interesting proposal for syntax that is being considered over on the docker repo, moby/moby#32100. Here they consider a very explicit EXPORT and IMPORT syntax. Just more food for thought.

[vsoch]

hey everyone I have a brain dump coming soon - need to decide where to put it :)

[blakedewey]

I'm glad my request has fueled such fantastic brain dumps so far! Looking forward to the next one!!

[vsoch]

Ok, this is what I think I'd want to do: https://gist.github.com/vsoch/ecd8386c8bd32936fb7574679a36b87c. If we create a simple approach like this that:

1. fits in with current software organization
2. is both modular for data and software, but still creates reproducible containers
3. allows for programmatic parsing to be able to easily find software and capture the contents of a container.

Then we can have a more organized base to work from, along with clearer directions (even templates) for researchers to follow to create software. We tell them how to build their thing, we package it in a way so it can be added to a container like:

import data://anaconda3:latest

The term "data" is some uri akin to docker or shub that would provide the packages. And that would cache that entire package for the user (for re-use) and also make it possible (if ever wanted) to export, not only from a bootstrap, but from a container:

bootstrap

singularity export container.img data://anaconda3:latest 

I would even go as far as to say that we stay away from system provided packages / software that are provided with package managers. For example, instead of a system python, I would use anaconda, miniconda, etc.

Anyway - please read the above. I think if we work on a simple goal for this standard, and then start building examples and tools around it, we can tackle a few problems at once: the organizational / curation issue with containers, the ability to have more modularity while still preserving reproducibility, and finally, a repository of data and software containers.

[vsoch]

BRAIN DUMP :hankey:

[boegel]

What's the current status of this?

Correct me if I'm wrong, but it's now possible to use an existing Singularity container image as a base for building a new image, is it not?

My question is triggered by HPCCM claiming that multi-stage builds aren't supported in Singularity, cfr. https://github.com/NVIDIA/hpc-container-maker#current-limitations

[GodloveD]

Oh man. Yes this is totally supported and has been for a long time. Not only is it possible to build from an existing image (local or hosted somewhere), it's also totally possible to run a definition file on top of an already built container. In this case, the header will be skipped and all sections will be run on the existing container.

[WestleyK]

Closing, because https://github.com/sylabs/singularity/pull/3156 is merged.