Docker Registry configuration

[hsiliev]

https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/docker_registry_configuration.md

is the proposal Docker Registry configuration arguments

[onsi]

@hsiliev - thanks for the proposal, I think this is a great way to communicate given the distance :)

Some thoughts:

• I prefer the SSL Validation Flag approach. I think we could have --skip-ssl-validation apply to both the private and public registries. That would resolve your point in the Con column. (I also think that supporting external insecure registries is not necessarily something we need to worry about at this point).
• You don't specify two things:
  1. How we specify -dockerRegistryURL -- is it a fixed IP? Is it a URL that we resolve using consul?
  2. How the stager will determine which egress rules to apply to the container in order to poke holes to the Docker registry.

I can propose a solution for both these things:

We register the private docker registry with Consul (we have examples of doing this with the file server). Then we would pass -dockerRegistryURL=http(s)://private_docker_registry.service.dc1.consul:PORT.

To open up the container I can imagine two options:

1. Stager fetches a list of all registered private_docker_registry services. This would return all registered IPs and ports and we could poke holes allowing access to all those IPs and ports.
2. We pick a unique port for the private docker registry that doesn't conflict with anything else in the VPC (hard/tricky!) and then open up that port on the entire VPC. I prefer option 1. There's a small race in that a new Docker registry may appear/disappear while we are staging. This would result in a staging failure but I imagine this would be very infrequent.

Thoughts?

[hsiliev]

@onsi: The registry client expects a []string with hosts that are insecure, but are allowed. Therefore docker builder should know upfront which hosts are insecure.

We can of course build the list dynamically and include the private and provided registry hosts when --skip-ssl-validation is specified. This disables certificate checks for the public docker registry as well, so Diego/CF gets vulnerable to attacks.

If we want to be on the safe side we might hard-code docker URL in the builder or at least add it as default value in the configuration, which seems a bad idea. Instead I went the other way around and decided to white-list the insecure hosts - the approach originally used by Docker client.

Consul solution looks nice, especially since we want to enable HA registry setup. I'll include your proposal. I would vote for 1, since the second option looks like a lot of pain for operators.

[onsi]

@hsiliev thanks for the update. I think I agree that --skip-ssl-validation should only apply to the private registry. We should only support secure external registries for now.

Let's go with option 1 for consul then. Great!

[hsiliev]

Added a new argument -consulAgentURL to:

• enable discovery of multiple registries
• allow host name to IP resolving for egress rules

[hsiliev]

Updated the proposal with:

• insecureDockerRegistries argument
• Egress rules
  • split to staging & running
  • implicitly adding rules for registry access

[onsi]

This all looks fine except for one thing: there is no need to add Egress rules when running. Garden downloads the DockerImage on the user's behalf (this does not happen in a container and, therefore, is not restricted by the egress rules). Only staging (which must fetch then re-upload the image inside a container) has this constraint.

[georgethebeatle]

Proposed opt-in config. Please review and comment.

[hsiliev]

Updated the stager flags as a result of 92988118:

• removed -dockerRegistryURL
• added -insecureDockerRegistry

Documented builder flag -dockerDaemonExecutablePath, used to specify the path to docker executable