Add/google build

[vsoch]

This is start of work to add Google Build, using a Docker Hub base image with Singularity as the builder. I will update this PR as I work on it, still figuring out some tweaks to get everything fully working.

[vsoch]

An update to the pull request - documentation is now finished and basic testing is done! A complete interaction can be seen in the following asciinema:

https://asciinema.org/a/227887?speed=2

Build Logic

As reviewed in #178, I will again walk through the steps that the builder takes. A more complete documentation is provided in the attached pdf (that will deploy to the site pages when the PR is merged).

google-build-client.pdf

Overall:

1. The user must export their application credentials and project id
2. The command is generally sregistry build <recipe> <context>, and context can be undefined (just the recipe) or a list of files, or all files in the $PWD (.)
3. The build runs and completes, honoring the users wishes to clean up the cloud bucket cache (with dependencies), make the container public (default), and add metadata to the container itself.
4. The final media link, and logging URL are printed to the screen.

Test Example

Create a folder "test" and add some Singularity file in it. If you have local filesystem dependencies, put them in this folder.

Bootstrap: docker
From: continuumio/miniconda3

%runscript
    exec echo "$@"

%post
    apt-get update

Then install the package from this branch

python setup.py install

and run the build. Note that we don't end with "." so we only need the Singularity recipe as a build context.

$ sregistry build --name vanessa/llama Singularity

If we had other dependencies to upload in the folder, you could do either:

$ sregistry build --name vanessa/llama Singularity .
$ sregistry build --name vanessa/llama Singularity file1 file2

Questions

Remaining questions for @wkharold !

Resource Allocation

The memory of the builder, along with the size of objects that can be handled (for example, a lot of scientific containers don't have significant bundles to upload, but download a lot of packages / source code to compile during build - how much space is default / allowed?) I did some basic testing, and with the builder a relatively simple file (just downloading docker layers) amounts to 182 seconds:

LOG Generating build package for 1 files...
LOG Uploading build package!
PROJECT singularity-static-registry0/0 MB - 00:00:00
BUILD   singularityware/singularity:3.0
LOG build 85c46ba8-6df0-4cf5-9134-0df7625b55cc: QUEUED
LOG build 85c46ba8-6df0-4cf5-9134-0df7625b55cc: WORKING
LOG build 85c46ba8-6df0-4cf5-9134-0df7625b55cc: WORKING
LOG build 85c46ba8-6df0-4cf5-9134-0df7625b55cc: WORKING
LOG build 85c46ba8-6df0-4cf5-9134-0df7625b55cc: WORKING
LOG build 85c46ba8-6df0-4cf5-9134-0df7625b55cc: WORKING
LOG build 85c46ba8-6df0-4cf5-9134-0df7625b55cc: WORKING
LOG build 85c46ba8-6df0-4cf5-9134-0df7625b55cc: WORKING
LOG build 85c46ba8-6df0-4cf5-9134-0df7625b55cc: WORKING
LOG build 85c46ba8-6df0-4cf5-9134-0df7625b55cc: WORKING
LOG build 85c46ba8-6df0-4cf5-9134-0df7625b55cc: WORKING
LOG build 85c46ba8-6df0-4cf5-9134-0df7625b55cc: WORKING
LOG build 85c46ba8-6df0-4cf5-9134-0df7625b55cc: SUCCESS
LOG Total build time: 182.7468774318695 seconds

(about 3 minutes). The same build on my local machine takes just 1 minute 20 seconds, and it was mostly downloading layers and then building the SIF (note I'm on crappy wireless, so on a reasonable HPC machine this would be much quicker)

Cleaning up...

real    1m20.301s
user    1m31.473s
sys 0m3.799s

[vsoch]

The basic functionality is complete, and we can tweak as needed. Going to merge and release.