docker / for-mac

Bug reports for Docker Desktop for Mac
https://www.docker.com/products/docker#/mac
2.43k stars 117 forks source link

Cannot create a multi-node swarm in Docker for Mac #67

Closed justinhj closed 8 years ago

justinhj commented 8 years ago

Expected behavior

Support for "docker swarm" commands in Docker for Mac so that one can create a multi-node swarm and use the new orchestration features of Docker 1.12

Actual behavior

Currently the swarm init command will register a swarm using the IP address of the Docker host running on the hypervisor which will not be accessible from another machine.

Information

Current version in use 1.12.0 build 81071

Steps to reproduce the behavior

Follow the steps in this tutorial but using Macs instead of VM's

https://docs.docker.com/engine/swarm/swarm-tutorial/

friism commented 8 years ago

@justinhj running docker swarm init works fine on Docker for Mac and Windows and you can use the new swarm-mode features (eg. service create).

If you want to test multi-node 1.12 swarms, I recommend using docker-machine (available in Docker for Mac) to create a couple VMs (or machines that you otherwise have access to) and form them into a swarm.

bastibense commented 8 years ago

I have this problem too, apparently.

When I create a new swarm using docker swarm init --advertise-addr 10.0.1.22 on my Mac (where 10.0.1.22 is an IP of the Mac itself), I'll get the following error:

Error response from daemon: advertise address must be an IP address or network interface (with optional port number)

When I run init without --advertise-addr, I'll get the following output:

Swarm initialized: current node (96nk3rf2baoracyrmilmjmzf7) is now a manager.

To add a worker to this swarm, run the following command: docker swarm join \ --token SWMTKN-1-5mdvml1wp7wzv318qymzqi8rgg6w6j3ma904nkrq7f33ndd0b4-d670pkto548caentg2aoehjkm \ 192.168.65.2:2377

The IP 192.168.65.2 apparently is the IP of the docker engine/hypervisor (not sure yet how the new architecture works in detail on OS X, yet).

untitled diagram

This IP unfortunately can't be accessed from outside the machine like OP said, from another machine on the network.

The second problem is, that I can't assign an IP from my machine to work around this, rendering the swarm commands rather useless in my case. And worse: the tutorials and documentation is not consistent with what actually happens.

Am I doing something wrong or is this a bug?

friism commented 8 years ago

@bastibense out of curiosity, what are you trying to accomplish with this? Just experimentation?

bastibense commented 8 years ago

@friism: Not sure what to tell you. ;) I'm just trying to follow the documentation. You could try to work through the tutorial yourself with docker for mac, and see the bug preventing you from doing what the tutorial says.

bastibense commented 8 years ago

https://docs.docker.com/engine/swarm/swarm-tutorial/

bastibense commented 8 years ago

Just came across this in the tutorial:

Note: Docker for Mac and Docker for Windows let you use single-node features of swarm mode, like creating a swarm and creating a service. Multi-node features like joining additional nodes and scaling a service are not available.

Is this refererencing this issue or something else?

friism commented 8 years ago

@londoncalling can we add a note to the swarm tutorial that it won't work with Docker for Mac and Windows being one of the nodes. I haven't thought through it, but I'd recommend the guide says to install Docker for Mac/Win, but then use docker-machine to create three separate hosts to test swarm on. Here's the relevant section: https://docs.docker.com/engine/swarm/swarm-tutorial/#/docker-engine-1-12-or-later

@bastibense as outlined above, I think you'd need to create three separate hosts using docker-machine and then wire them up into a swarm that's separate from the Docker for Mac system.

justinhj commented 8 years ago

@friism personally I was just experimenting but I think using Docker for Mac would be a much better experience than docker-machine when you want to show someone how to use swarm on multiple physical machines.

As noted in later comments it would be great if the swarm tutorial was clear that you have to use docker-machine and that it is safe to do so. As a use of the Docker for Mac Beta I often ended up in a messed up state because of running different docker-machine versions and so on. That made me wary to try to use both at once.

I think even in production environments people would like to be able to connect a local laptop to a swarm cluster. For example Weaveworks lets you do this.

I mentioned my problems in twitter exchange with Justin Cormack who recommended opening this ticket.

londoncalling commented 8 years ago

Hi, I can definitely add a docs section on this. I worked through the swarm tutorial last week or so using Docker Machine. I have also gotten in a state where my versions don't agree (I think it's when I upgrade Docker for Mac, then try to manage nodes I created with an older version of Machine and / or Engine ?), and despite all that stuff we wrote on the topic of version management in "Toolbox v. Docker for Mac", the only way I know how to get out of it is to recreate the nodes with a new version of Machine.

Justin when you say you think experimenting with Swarm using Docker for Mac would be better , do you mean a) using the version of Machine that comes with d4mac, or b) making d4mac capable of creating multiple nodes , like Machine can do? --Vicky

On Wednesday, August 24, 2016, Justin Heyes-Jones notifications@github.com wrote:

@friism https://github.com/friism personally I was just experimenting but I think using Docker for Mac would be a much better experience than docker-machine when you do want to experiment with swarm on multiple physical machines.

As noted in later comments it would be great if the swarm tutorial was clear that you have to use docker-machine and that it is safe to do so. As a use of the Docker for Mac Beta I often ended up in a messed up state because of running different docker-machine versions and so on. That made me wary to try use both.

I think even in production environments people would like to be able to connect a local laptop to a swarm cluster. For example Weaveworks lets you do this.

I mentioned my problems in twitter exchange with Justin Cormack who recommended opening this ticket.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/docker/for-mac/issues/67#issuecomment-242131350, or mute the thread https://github.com/notifications/unsubscribe-auth/ALHuA5Uv6uvVhmuAVSEOkXWGNJ3LDDjTks5qjHSBgaJpZM4JaQn8 .

friism commented 8 years ago

@londoncalling for now, it would be a), using the docker-machine that comes with Docker for Mac.

The API version problem is painful, it's being addressed here: https://github.com/docker/docker/issues/25498

For now, we could perhaps recommend people set DOCKER_API_VERSION env var to work around it.

londoncalling commented 8 years ago

@friism cool, I'll add a section more specifically on Swarm to the docs, and maybe you guys can review and we'll go from there.

justinhj commented 8 years ago

Hi @londoncalling

Justin when you say you think experimenting with Swarm using Docker for Mac would be better , do you mean a) using the version of Machine that comes with d4mac, or b) making d4mac capable of creating multiple nodes , like Machine can do? --Vicky

I think ideally swarm would work seamlessly with a d4mac host and no docker machine. But since that sounds tricky and may be for some time in the future, I think explaining to users how to work around it is a good stopgap.

londoncalling commented 8 years ago

@friism I think it would be worth us getting together to write this, maybe tomorrow or the next day when I'm in the office. I have several questions regarding networking, discovering IP addresses for nodes as well as containers on w/d4mac, d4win, that I'd like to talk through, along with really understanding the current workarounds to the version thing, includingDOCKER_API_VERSION env var. With that view, I'll be able to add better instructions in both the Swarm Tutorial (owned by Charles) as well as in d4mac, d4win docs. We can get this done and published same day.

friism commented 8 years ago

Alright, the below works on Windows.

Prerequisite is Hyper-V switch (here called Better New Virtual Switch). All of this is running in Administrator Powershell.

Get machines going:

> docker-machine create -d hyperv --hyperv-virtual-switch "Better New Virtual Switch" master
> docker-machine create -d hyperv --hyperv-virtual-switch "Better New Virtual Switch" worker1
> docker-machine create -d hyperv --hyperv-virtual-switch "Better New Virtual Switch" worker2

Init swarm:

> docker-machine inspect --format '{{ json .Driver.IPAddress }}' master
"192.168.202.112"
> docker-machine ssh master docker swarm init --advertise-addr 192.168.202.112
To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-4k5ljcmxs1d9q14lth4tfbg868lf8eqi5alxtvgo7s1ptyrhlu-3ihz3bfmx5622vei1smzetudf \
    192.168.202.112:2377

Add the workers:

> docker-machine ssh worker1 docker swarm join --token SWMTKN-1-4k5ljcmxs1d9q14lth4tfbg868lf8eqi5alxtvgo7s1ptyrhlu-3ihz3bfmx5622vei1smzetudf 192.168.202.112:2377
> docker-machine ssh worker2 docker swarm join --token SWMTKN-1-4k5ljcmxs1d9q14lth4tfbg868lf8eqi5alxtvgo7s1ptyrhlu-3ihz3bfmx5622vei1smzetudf 192.168.202.112:2377

SSH into the master and go to town (or use it from the host):

> docker-machine ssh master
> docker node ls
 ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
 aojoo2h0uuj5hv1c9xajo67o2    worker1   Ready   Active
 eqt1yd8x52gph3axjkz8lxl1z *  master    Ready   Active        Leader

(I had three nodes)

friism commented 8 years ago

@londoncalling btw, we also have these: https://github.com/docker/labs/tree/master/swarm-mode

londoncalling commented 8 years ago

@friism , okay thanks! I'm working through Docker labs stuff and re-writing some of it so I'll have a look at the Swarm topics there next.

Also, here is the output of a Swarm walkthrough I did the other day w/Docker Machine using Digital Ocean cloud nodes: my swarm tutorial output.

I guess the only thing I was confused about was users trying to do this with d4mac, d4win only, some networking questions, and version incompatibilities that can come up.

If we are just talking about using the Machine that comes with d4mac, d4win, then I can go ahead and include an abbreviated Swarm tutorial right in the Piñata docs and/or link out and add the caveats we talked about.

londoncalling commented 8 years ago

@friism @justinhj @bastibense this is addressed now in docs per this PR: https://github.com/docker/docker/pull/26028. Should be published at end of day or early next week.

londoncalling commented 8 years ago

@friism @justinhj @bastibense fyi, docs are now updated properly per https://github.com/docker/docker/pull/26028. You can see them as detailed below. Next step is to test and add links to Mano's labs examples: https://github.com/docker/labs/tree/master/swarm-mode, tbd this week.

Details in the Swarm tutorial, especially this topic on how to get Docker Engine or newer.

The Swarm docs are linked to directly from d4mac, d4win docs in "what's next" and FAQ's sections:

friism commented 8 years ago

Can this be closed?

londoncalling commented 8 years ago

@friism yes, thanks! I can close it.

londoncalling commented 8 years ago

FYI, just to close the loop here, the docs are now updated with details on how to use the Hyper-V driver for Docker Machine, and how to use Docker Machine with Docker for Mac, Docker for Windows. See Get started with Docker Machine on a local VM and Microsoft Hyper-V driver for Docker Machine

justinhj commented 8 years ago

This is great. Thanks for your help everyone.

londoncalling commented 8 years ago

Thanks Justin - Well you still might hit some problems on Docker for Windows. Let me know if you do. Creating machines on Windows with hyper-v driver seems to work variably and hang sometimes at either waiting for SSH or before.

I should have an update for those docs again this week hopefully. In the meantime , feel free to send more feedback !

Cheers, Vicky

On Tuesday, September 27, 2016, Justin Heyes-Jones notifications@github.com wrote:

This is great. Thanks for your help everyone.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/docker/for-mac/issues/67#issuecomment-250029864, or mute the thread https://github.com/notifications/unsubscribe-auth/ALHuA1uj89ge661RN7duVs2FVhAIfhw6ks5quagfgaJpZM4JaQn8 .

londoncalling commented 8 years ago

@justinhj all is working for me now on Docker for Windows Beta 27, and for @ManoMarks on the latest stable Beta (v1.12.1) . Let us know if you run into more problems. Thanks!

shkrid commented 7 years ago

Here the another solution http://blog.terranillius.com/post/swarm_dind/ of this problem. Use dind for workers

londoncalling commented 7 years ago

@shkrid thanks, we do mention that you can work with a swarm on a single node here but we don't go into the detail you did. We might try to incorporate your example at some point if that's okay.

docker-robott commented 4 years ago

Closed issues are locked after 30 days of inactivity. This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle locked