search

coreos / bugs

Issue tracker for CoreOS Container Linux
https://coreos.com/os/eol/
147 stars 30 forks source link

proposal: allow --insecure-registry to the private network bridge for docker v1.3.2 #594

Closed crawford closed 9 years ago

crawford commented 9 years ago

Issue by bacongobbler Thursday Nov 13, 2014 at 01:39 GMT Originally opened as https://github.com/coreos/coreos-overlay/issues/960

Docker v1.3.1 introduced the notion of --insecure-registry <ip addr>. Whether this was the right or wrong thing to do is out of the scope of this discussion. CoreOS rolled back to docker v1.3.0 for the time being (thank goodness), but when CoreOS eventually updates to docker v1.3.2, I'd like to propose that we should whitelist any HTTP docker registries running across the cluster by passing --insecure-registry=10.0.0.0/8 or w/e is the sane default for allowing us to use registries across the private network.

To explain our use case, pre-v1.3.1 we ran a CoreOS cluster with a HTTP docker registry scheduled by fleet and it published itself to etcd so other components could push images to the registry. We do not require a registry backed by SSL because we are running it completely internal to the cluster and nobody outside has access to it, so the notion of tacking on SSL for the sake of docker client compatibility is insane. With v1.3.1, you cannot push to an insecure registry unless you told the daemon the IP address that should be whitelisted. This poses an issue because this is configured at boot time. If you add a new node to the cluster and the registry is scheduled there due to a node failure or a reboot, Your cluster suddenly cannot push to the registry any more. With v1.3.2, docker will be allowing subnets to be passed to --insecure-registry (docker/docker#9100).

Since we know that our private network is to be trusted, can we please add this option to the daemons?

crawford commented 9 years ago

Comment by mboersma Thursday Nov 13, 2014 at 17:19 GMT

+1 for this idea, as it wouldn't require manual reconfiguration of the Docker daemon.

crawford commented 9 years ago

Comment by crawford Thursday Nov 13, 2014 at 17:47 GMT

The problem with 10.0.0.0/8 in particular is the fact that not all environments use that /8 for their private range. Is there any downside to having the user write a drop-in via their cloud-init that specifies their particular trusted range?

crawford commented 9 years ago

Comment by bacongobbler Thursday Nov 13, 2014 at 18:05 GMT

The downside would mean that the user must manage their docker options on CoreOS updates. For the most part we're relying on the auto-updater to ensure that channel updates manage these option flags for us in case something changes in CoreOS that requires a new docker flag. Manual intervention with the flags is more of an issue with software backwards compatibility in general rather than something that needs to be patched downstream, though.

Is there a way for us to figure out the private range at boot such that we can staunch the wound for now until we find a solution? I do agree that this is a little tedious to bring into CoreOS due to how users customize their stack. If there's a generic "fits most buckets" solution that works on most systems and write up documentation for others that would help go a long way :)

crawford commented 9 years ago

Comment by crawford Thursday Nov 13, 2014 at 18:10 GMT

Yeah, this would be problematic for those updating to docker 1.3.1 (who wouldn't have that drop-in in place) and at some point in the future when we tweak the docker daemon flags (which would be overridden by the drop-in). That was a bad suggestion. :)

crawford commented 9 years ago

Comment by bacongobbler Thursday Nov 13, 2014 at 18:44 GMT

If this is hard to solve upstream, maybe there could be a way we can send in extra docker flags via cloudinit, possibly by setting something in /etc/environment? e.g. something like

write_files:
  - path: /etc/environment
    content: |
        COREOS_PUBLIC_IPV4=$public_ipv4
        COREOS_PRIVATE_IPV4=$private_ipv4
        DOCKER_OPTS="--insecure-registry 10.0.0.0/8"

That may help us with the general use case (cloud providers) and then we can document it for users on bare metal.

crawford commented 9 years ago

Comment by bacongobbler Monday Nov 17, 2014 at 18:39 GMT

I did not notice this before: https://github.com/coreos/coreos-overlay/blob/01e6f6d7c3179f558f6c271b72128eddea76be38/app-emulation/docker/files/docker.service#L10

Should we close this issue in favour of documentation around adding the following source code to our users' cloudinit files, modifying the bridge as needed or should we try to solve this in coreos-overlay? I'd be happy to supply patches as required :)

write_files:
  - path: /run/docker_opts.env
    content: |
        DOCKER_OPTS="--insecure-registry 10.0.0.0/8"
crawford commented 9 years ago

Comment by marineam Sunday Nov 23, 2014 at 19:45 GMT

I don't think that docker_opts.env is something we should officially document, it was added as a hook point for experimental automatic configuration of docker networking. @crawford and I discussed it a little and we think we will continue to support $DOCKER_OPTS in the environment, but via a standard dropin instead:

#cloud-config

write_files:
  - path: /etc/systemd/system/docker.service.d/50-insecure-registry.conf
    content: |
        [Service]
        Environment=DOCKER_OPTS="--insecure-registry 10.0.0.0/8"
crawford commented 9 years ago

Comment by bacongobbler Monday Nov 24, 2014 at 22:42 GMT

Is this possible today, given that 509.1.0 is out?

crawford commented 9 years ago

Comment by crawford Monday Nov 24, 2014 at 22:47 GMT

@bacongobbler yes, this technique should work.

crawford commented 9 years ago

Comment by bacongobbler Monday Nov 24, 2014 at 23:13 GMT

As an FYI, the CLI parser didn't like spaces:

core@deis-2 ~ $ journalctl -u docker
-- Logs begin at Mon 2014-11-24 23:03:29 UTC, end at Mon 2014-11-24 23:05:33 UTC. --
Nov 24 23:03:38 deis-2 systemd[1]: [/etc/systemd/system/docker.service.d/50-insecure-registry.conf:2] Invalid environment assignment, ignoring: DOCKER_OPTS="--insecure-registry 10.0.0.0/8"
Nov 24 23:03:38 deis-2 systemd[1]: [/etc/systemd/system/docker.service.d/50-insecure-registry.conf:2] Invalid environment assignment, ignoring: DOCKER_OPTS="--insecure-registry 10.0.0.0/8"
Nov 24 23:04:27 deis-2 systemd[1]: Starting Docker Application Container Engine...
Nov 24 23:04:27 deis-2 systemd[1]: Started Docker Application Container Engine.
Nov 24 23:04:27 deis-2 docker[1023]: flag needs an argument: --insecure-registry

The following fixed it up:

#cloud-config

write_files:
  - path: /etc/systemd/system/docker.service.d/50-insecure-registry.conf
    content: |
        [Service]
        Environment=DOCKER_OPTS='--insecure-registry="10.0.0.0/8"'

Closing as this solves our use case for now until we find a "better" solution. @marineam @crawford thank you for all your help

crawford commented 9 years ago

Comment by insanity54 Saturday Feb 14, 2015 at 11:38 GMT

Just had the spaces issue in 557.2.0. I need more than one docker option so the space is unavoidable:

/docker.service.d/50-media-srv-dir.conf:2] Invalid environment assignment, ignoring: DOCKER_OPTS='--insecure-registry="10.0.1.0/24" --graph="/media/srv/docker"'
crawford commented 9 years ago

Comment by insanity54 Saturday Feb 14, 2015 at 12:04 GMT

Nevermind. Found this issue containing a solution which made all my wildest dreams come true https://github.com/coreos/coreos-overlay/issues/1052