Not sure how to login to private registry

[jschneiderhan]

I'm trying to use mesos/marathon/deimos with an image hosted on a private registry, quay.io in my case. When I try to run the app, it fails with the following error:

Jul  2 19:09:38 vagrant deimos[1695]: deimos.containerizer.docker.launch()
Jul  2 19:09:38 vagrant deimos[1695]: deimos.state.lock() request // launch EX (60s)
Jul  2 19:09:38 vagrant deimos[1695]: deimos.state.lock() success // launch EX (60s)
Jul  2 19:09:38 vagrant deimos[1695]: deimos.containerizer.docker.launch() eid    = testapp-8_0-1404328178191
Jul  2 19:09:38 vagrant deimos[1695]: deimos.containerizer.docker.launch() mesos  = fdd21297-c501-4626-a757-d299e3f3097d
Jul  2 19:09:38 vagrant deimos[1695]: deimos.containerizer.docker.launch() image  = quay.io/myuser/testapp:latest
Jul  2 19:09:38 vagrant deimos[1695]: deimos.state.lock() request // observe EX|NB (60s)
Jul  2 19:09:38 vagrant deimos[1695]: deimos.state.lock() success // observe EX|NB (60s)
Jul  2 19:09:39 vagrant deimos[1695]: deimos.docker.pull() exit 1 // docker pull quay.io/myuser/testapp:latest
Jul  2 19:09:39 vagrant deimos[1695]: deimos.docker.pull() STDERR // 2014/07/02 19:09:39 HTTP code: 403
Jul  2 19:09:39 vagrant deimos[1695]: deimos.cli() Command '['docker', 'pull', u'quay.io/myuser/testapp:latest']' returned non-zero exit status 1

Which makes sense. My login shell can pull from the registry because I run docker login. which cached my credentials in ~/.dockercfg. It doesn't look like docker login is executed here, and even if it did my credentials have not been passed in.

Are there any known ways to allow deimos to authenticate with a private registry?

If I can help debug in any way let me know. I'm loving mesos/marathon/deimos so far!

[solidsnack]

Maybe copying ~/.dockercfg to /root/.dockercfg would work? Deimos runs as root, directly under the slave. There doesn't seem to be much documentation on how to control which .dockercfg file is used for authentication; the name of the file itself turns up only one hit in the Docker docs:

https://www.google.com/#q=site:docs.docker.com+.dockercfg

[jschneiderhan]

@solidsnack thank you for getting back to me. I'm pretty sure I tried copying the file over to /root/.dockercfg and it didn't work. I'll confirm today and get back to you.

Is supporting authenticating with a private registry something that you would like to support in deimos directly by calling the docker 'login' command prior to pull, as opposed to relying on the presence of a .dockercfg file? If so, I'd be happy to take a shot at adding it.

[jschneiderhan]

No dice when copying the file to /root/.dockercfg, still get a 403 when pulling.

Just for the hell of it, I hardcoded a "docker login" call right before the pull in docker.py and did a find for the .dockercfg file. It ended up being placed in:

/tmp/mesos/slaves/20140703-141945-16842879-5050-9288-0/frameworks/20140703-141945-16842879-5050-9288-0000/executors/ubuntu_0-1404399606067/runs/15363f63-5689-49e2-8038-fc5d45ab34d4/.dockercfg

[solidsnack]

It'd be great to have support for it. Given what you've discovered, I'm really not sure what the best way to add it is -- logging in every time seems like a bad pattern -- but I'd be happy to accept what you come up with.

Maybe the right thing to do is have Deimos copy a specific .dockercfg into .?

[jschneiderhan]

@solidsnack that would do the trick. The only way that I can think of providing the contents of the .dockercfg file to deimos is via an attribute in deimos.cfg. Perhaps base64 encoding a valid dockercfg file and providing it as a setting? When the executor sets up the working directory deimos can write the contents to the file. The .dockercfg file format allows the specification of credentials for multiple registries, so a global value for the dockercfg files should be able to hold login info for all of the containers expected to be deployed across the cluster.

Does that sound like an acceptable approach? It's definitely a kludgy solution but the authentication interface with docker doesn't leave a while lot of options.

[solidsnack]

Would it not be better to simply configure Deimos with a path to the .dockercfg file?

[docker.index]
dockercfg = /etc/dockercfg

Base64 encoding the file and inlining it would seem to have at the least the disadvantage that one is forced to treat the Deimos configuration file as a holder of secure credentials.

[jschneiderhan]

Yes, that would be better.

[solidsnack]

Haven't tested the code in 73b88728ca85a1762e8c63772f241ab21d560a46 but it's a start. Could you look it over and maybe give it a shot?

[jschneiderhan]

@solidsnack I just tried out that commit but still received a 403. It looks like the docker pull is being executed prior to the dockercfg being placed. I traced it down to somewhere is this block: https://github.com/mesosphere/deimos/blob/dockercfg/deimos/docker.py#L20-L27 . Their is a comment which says "Forces external call to pre-fetch image".

I tried moving the 'self.place_dockercfg()' call right before https://github.com/mesosphere/deimos/blob/dockercfg/deimos/containerizer/docker.py#L123, but https://github.com/mesosphere/deimos/blob/dockercfg/deimos/containerizer/docker.py#L328 is evaluating to false, so the config file is not set.

[solidsnack]

There was a bug in the config loader.

I've pushed a new branch with your changes and the fix to the config file reader.

diff --git a/deimos/config.py b/deimos/config.py
index 097a1f2..ab58a1e 100644
--- a/deimos/config.py
+++ b/deimos/config.py
@@ -189,6 +189,9 @@ def parse(f):
         del parsed["containers.options"]
     if len(containers) > 0:
         parsed["containers"] = Containers(**containers)
+    if "docker.index" in parsed:
+        parsed["index"] = parsed["docker.index"]
+        del parsed["docker.index"]
     return _Struct(**parsed)

diff --git a/deimos/containerizer/docker.py b/deimos/containerizer/docker.py
index ac1155c..3b750a6 100644
--- a/deimos/containerizer/docker.py
+++ b/deimos/containerizer/docker.py
@@ -120,14 +120,14 @@ class Docker(Containerizer, _Struct):
         else:
             env += mesos_env() + [("MESOS_DIRECTORY", self.workdir)]

+        self.place_dockercfg()
+
         runner_argv = deimos.docker.run(run_options, image, launchy.argv,
                                         env=env, ports=launchy.ports,
                                         cpus=cpus, mems=mems)

         log_mesos_env(logging.DEBUG)

-        self.place_dockercfg()
-
         observer = None
         with open("stdout", "w") as o:        # This awkward multi 'with' is a
             with open("stderr", "w") as e:    # concession to 2.6 compatibility

[jschneiderhan]

Just tried and received the following error:

Jul  8 18:24:38 vagrant deimos[4456]: deimos.containerizer.docker.launch()
Jul  8 18:24:38 vagrant deimos[4456]: deimos.state.lock() request // launch EX (60s)
Jul  8 18:24:38 vagrant deimos[4456]: deimos.state.lock() success // launch EX (60s)
Jul  8 18:24:38 vagrant deimos[4456]: deimos.containerizer.docker.launch() eid    = ubuntu_0-1404843876386
Jul  8 18:24:38 vagrant deimos[4456]: deimos.containerizer.docker.launch() mesos  = 50de9b28-f781-481e-a14d-8bcb2e0c54f2
Jul  8 18:24:38 vagrant deimos[4456]: deimos.docker.matching_image_for_host() call // bash -c '#012            set -o errexit -o nounset -o pipefail#012            ( source /etc/os-release && tr A-Z a-z <<<"$ID#011$VERSION_ID" )#012        '
Jul  8 18:24:38 vagrant deimos[4456]: deimos.docker.matching_image_for_host() exit 0 // bash -c '#012            set -o errexit -o nounset -o pipefail#012            ( source /etc/os-release && tr A-Z a-z <<<"$ID#011$VERSION_ID" )#012        '
Jul  8 18:24:38 vagrant deimos[4456]: deimos.docker.matching_image_for_host() STDOUT // ubuntu#01114.04
Jul  8 18:24:38 vagrant deimos[4456]: deimos.cli() Unhandled failure in launch#012Traceback (most recent call last):#012  File "/srv/deimos/deimos/__init__.py", line 73, in cli#012    result = deimos.containerizer.stdio(containerizer, *argv[1:])#012  File "/srv/deimos/deimos/containerizer/__init__.py", line 97, in stdio#012    return method(recordio.read(proto), *args[1:])#012  File "/srv/deimos/deimos/containerizer/docker.py", line 72, in launch#012    image = self.determine_image(url, launchy)#012  File "/srv/deimos/deimos/containerizer/docker.py", line 314, in determine_image#012    default = self.image_from_system_context(launchy)#012  File "/srv/deimos/deimos/containerizer/docker.py", line 324, in image_from_system_context#012    return deimos.docker.matching_image_for_host(**opts)#012  File "/srv/deimos/deimos/docker.py", line 123, in matching_image_for_host#012    return image_token("%s:%s" % (distro, release), *args, **kwargs)#012TypeError: image_token() got an unexpected keyword argument 'dockercfg'
Jul  8 18:24:38 vagrant mesos-slave[4354]: E0708 18:24:38.612555  4371 slave.cpp:2310] Container '50de9b28-f781-481e-a14d-8bcb2e0c54f2' for executor 'ubuntu_0-1404843876386' of framework '20140703-141945-16842879-5050-9288-0000' failed to start: Could not launch container '50de9b28-f781-481e-a14d-8bcb2e0c54f2': External containerizer failed (status: 8)

I'm going to try and debug now ...

[solidsnack]

@jschneiderhan How is this working out for you today?

[jschneiderhan]

I just tried 95a7677, which is HEAD of the index-config-fixes branch, and it worked perfectly

[solidsnack]

This made it in to 0.4.0

[jschneiderhan]

Great. This will be super-useful for people using a private registry. I'm going to close the issue as my use case is now met. Thanks @solidsnack!