search

docker / machine

Machine management for a container-centric world
https://docs.docker.com/machine/
Apache License 2.0
6.62k stars 1.97k forks source link

Latest docker release (20.10.0) doesn't work with docker-machine #4858

Open joelgriffith opened 3 years ago

joelgriffith commented 3 years ago

Hey there! I've noticed that the latest docker release (20.10.0 as of this writing) breaks with docker-machine's create phase. The following no longer works:

$ docker-machine create --driver digitalocean --digitalocean-access-token REDACTED --engine-storage-driver overlay2 --digitalocean-image ubuntu-18-04-x64 --digitalocean-size s-1vcpu-1gb --digitalocean-region sfo2 dm-create-test
Running pre-create checks...
Creating machine...
(dm-create-test) Creating SSH key...
(dm-create-test) Creating Digital Ocean droplet...
(dm-create-test) Waiting for IP address to be assigned to the Droplet...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with ubuntu(systemd)...
Installing Docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Error creating machine: Error running provisioning: Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded

This has failed for about the last few hours. Seems like pinning to a custom engine URL works, but I imagine most folks will use this with a broken experience across all drivers. I've currently worked around this by setting:

--engine-install-url "https://releases.rancher.com/install-docker/19.03.9.sh"
joelgriffith commented 3 years ago

https://get.docker.com/ is the default install URL, fwiw

demahum commented 3 years ago

Interestingly, I just tried with::

docker-machine create --driver google --google-project blablabla --google-machine-image https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/ubuntu-1804-bionic-v20201201 anyname

And it worked fine.

After doing docker-machine ssh anyname, I get:

sudo docker version
Client: Docker Engine - Community
 Version:           20.10.0
...
Server: Docker Engine - Community
 Engine:
  Version:          20.10.0

I tried with 18, 19 and 20 local Docker versions. Docker machine is 0.16.0.

Do you see anything I'm missing (except that you used DO and I used GCP)?

joelgriffith commented 3 years ago

Thanks for checking this out. I haven't tried with other providers, just DO, so the problem might be there. Will check again later

joelgriffith commented 3 years ago

Looks like, according to https://github.com/JonasProgrammer/docker-machine-driver-hetzner/issues/54, that Hetzner doesn't work as well.

cardoe commented 3 years ago

Interestingly, I just tried with::

docker-machine create --driver google --google-project blablabla --google-machine-image https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/ubuntu-1804-bionic-v20201201 anyname

And it worked fine.

After doing docker-machine ssh anyname, I get:

sudo docker version
Client: Docker Engine - Community
 Version:           20.10.0
...
Server: Docker Engine - Community
 Engine:
  Version:          20.10.0

Querying the docker version will work but attempt to run a container on that box now and it will fail.

demahum commented 3 years ago

@cardoe, not really:

$ docker-machine ssh anothertest
Welcome to Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1030-gcp x86_64)
...
Last login: Wed Dec  9 18:10:01 2020 from 34.90.202.113
docker-user@anothertest:~$ sudo docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

docker-user@anothertest:~$
pierotofy commented 3 years ago

Confirm, latest update broke docker-machine create.

lcampanis commented 3 years ago

Confirm as well

xandrox commented 3 years ago

Me too, my workaround for gitlab runner and AWS driver was also to use engine-install-url with a fixed version script.

sebastien-gallet commented 3 years ago

docker-machine -D create ... --driver ... -D enable debug mode, so we can see where it breaks.

In the case of rackspace (and I suspect digitalocean as well), we discovered that the system was not allowing external connections to the docker service because the firewall was blocking the port 2376.

As a workaround we created an image with an additional firewall rule to allow 2376 on an external interface. Using that image, we were able to create a machine successfully.

Best would be to have the provision files updated with the proper fixes.

GeorgFleig commented 3 years ago

In my case (Gitlab runner at DigitalOcean, Ubuntu 20.04 LTS) the update to Docker 20.10.0 broke the runners as well. After a bit of debugging it seemed to me as if the docker daemon was only listening on the socket, not the port, when docker-machine started checking for the running docker daemon. According to the logs, half a minute after docker-machine gave up (10 retries), the docker daemon was restarted and also listened on port 2376.

I connected locally to the docker daemon and was able to run containers.

Haven't investigated further and just changed to Docker 19.03 with the parameter suggested bei @joelgriffith which fixes the issue for now.

thaJeztah commented 3 years ago

This may be related to https://github.com/moby/moby/issues/41767 (systemctl docker start get stuck in cloud-init), for which we'll be publishing updated packages

cardoe commented 3 years ago

@cardoe, not really:


$ docker-machine ssh anothertest

Use eval $(docker-machine env anothertest) and you'll see it's broken. You need to connect the same way that the rest of the system will connect. by SSHing in you're using the UNIX socket. Attempting to do this via the TCP socket will show you the issue.

afbjorklund commented 3 years ago

As far as I know docker-machine is deprecated, and won't be updated anymore: see https://github.com/docker/machine/issues/4537

There will be no further releases of the boot2docker.iso either: https://github.com/boot2docker/boot2docker/pull/1408

So Docker 19.03 is the final release, unless Docker Inc. changes their minds about the projects...

We will have a fork of libmachine inside minikube, but the machine-drivers will not be compatible.

afbjorklund commented 3 years ago

In the case of rackspace (and I suspect digitalocean as well), we discovered that the system was not allowing external connections to the docker service because the firewall was blocking the port 2376.

I have done some work for minikube, to switch from the old tcp to the new ssh connections.

This could be ported to docker-machine, if the project finds a new life somewhere else ? Like moby-machine and boot2moby, or whatever. Something that is accepting patches...

Basically it uses ssh:// for the docker host, but also need to use ssh-add for the keys.

Miouge1 commented 3 years ago

I think I ran into this problem. I get the following error:

Error creating machine: Error running provisioning: Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded

My investigation shows that the docker daemon started on the target machine, listens only on the docker socket.

A restart of the docker daemon "at the right" time allows the reload the config and listen on the expected port.

Until v20.10 supported is fixed, I've added a VERSION=19.03 in the --engine-install-url. I suppose using the rancher URL would work as well.

For people running gitlab-runner with docker+machine executor, what is frustrating is that there is no error in the docker-machine ls output.

webflo commented 3 years ago

@Miouge1 Whats the syntax for engine-install-url? Could you provide a complete example? Thanks!

Nothing4You commented 3 years ago

@webflo there's an example in the first post in this issue

thaJeztah commented 3 years ago

Docker 20.10.1 packages are now available on download.docker.com, and may solve this issue; perhaps someone could give it a test to verify if the issue is resolved?

randomshinichi commented 3 years ago

Docker 20.10.1 has the same problem - docker-machine writes the systemd unit file, calls daemon-reload, but for some reason ps ax |grep dockerd shows that the running dockerd has a blank -H argument. If you ssh in and restart docker manually, it comes up with the correct arguments.

tsnowlan commented 3 years ago

This could be ported to docker-machine, if the project finds a new life somewhere else ? Like moby-machine and boot2moby, or whatever. Something that is accepting patches...

Gitlab has a fairly active fork that releases fixes for various issues: https://gitlab.com/gitlab-org/docker-machine

It's one of the main executors for their test runners and it's still having this issue, so hopefully they'd accept a PR with a fix.

afbjorklund commented 3 years ago

@tsnowlan : as far as I know, both https://github.com/machine-drivers/machine and https://gitlab.com/gitlab-org/ci-cd/docker-machine are for bugfixes only (not for development)

https://docs.gitlab.com/runner/executors/docker_machine.html#forked-version-of-docker-machine

The intent of this fork is to fix critical and bugs affecting running costs only. No new features will be added.

Things like using ssh transport for docker or adding support for ssh host keys are not really bug fixes.

So that is why it needed to start over, outside Docker. For now, just using Vagrant instead of "machine"...

I would be happy to contribute to a replacement for docker-machine / podman-machine, but won't really drive. It is clear that neither of the upstream companies wants anything to do with the open source projects anymore.

tsnowlan commented 3 years ago

Ah, yeah. I suppose that while those could fix some issues they aren't really bug fixes themselves.

It's pretty frustrating. Gitlab just had a docker-machine release a week ago, after this problem had been reported and 20.10.1 released, and it's still not working.

XedinUnknown commented 3 years ago

Hi all! I read everything, but not sure if a workaround exists at this point. Trying to create a droplet on DigitalOcean, but no joy. Could you confirm, and if a workaround exists - make it clear please? This would really help, because I imagine that a lot of people would find this thread soon

Nothing4You commented 3 years ago

@XedinUnknown read again, a workaround is literally in the first post

XedinUnknown commented 3 years ago

It appears to be talking about an arg, but I don't know what this is an arg for.

Nothing4You commented 3 years ago

docker-machine, the very thing this issue is about

baxeico commented 3 years ago

I confirm that this command works:

docker-machine create --driver digitalocean --digitalocean-image ubuntu-20-04-x64 --digitalocean-access-token $DOTOKEN --digitalocean-region fra1 --engine-install-url "https://releases.rancher.com/install-docker/19.03.9.sh" docker-ubuntu-2004

paulbygrave commented 3 years ago

Thanks for flagging this issue and raising a quick fix using the --engine-install-url flag. We're using this as the final straw and migrating off docker-machine ASAP, it's just causing us too much of a maintenance headache.

rajaie-sg commented 3 years ago

What's interesting is that I only run into this issue when I use an AMI that already has Docker v20.10.2 installed. If I let docker-machine install Docker, it installs v20.10.2 and seems to work fine though.

martinsotirov commented 3 years ago

I might be missing something but docker-machine works perfectly fine for me with the latest Docker and the hetzner cloud provider.

szEvEz commented 2 years ago

@martinsotirov Could you share your config.toml? Because with the latest Docker version and the Hetzner Provider I am still running into these issues

qwe@167dc61c5bc5:/# docker-machine ls
NAME                                             ACTIVE   DRIVER    STATE     URL                         SWARM   DOCKER     ERRORS
runner-vfpp49y5-testrunner-1626776429-af5d8edb   -        hetzner   Running   tcp://117.204.20.159:2376           v20.10.7 
docker logs root_hetzner-runner_1

ERROR: Error creating machine: Error running provisioning: Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded  driver=hetzner name=runner-vfpp49y5-testrunner-1626776429-af5d8edb operation=create
martinsotirov commented 2 years ago

@szEvEz I don't have my Hetzner Swarm running anymore but as far as I remember I used the guide from the Hetzner driver. After initializing it wasn't able to find the nodes but after restarting them, it just worked.

I'm gonna try to recreate a swarm soon and I'll write here how it goes.

sebvaldez commented 2 years ago

EDIT: This command worked for me:

$ docker-machine create \
   --driver digitalocean \
   --digitalocean-image ubuntu-20-04-x64 \
   --digitalocean-access-token $DOTOKEN \
   --digitalocean-region sfo3 \
   --engine-install-url "https://releases.rancher.com/install-docker/19.03.9.sh"
   boomi

Hello all, I too am having troubles creating a docker-machine with digital ocean. any advice?

$ docker-machine -v
  => docker-machine version 0.16.0, build 702c267f
$ docker -v 
  => Docker version 20.10.7, build f0df350
docker-machine -D create \
    --driver digitalocean \
    --digitalocean-access-token $DOTOKEN \
    --digitalocean-region "sfo3" \
    --digitalocean-image 85785636 \
    --digitalocean-size "s-2vcpu-2gb" \
    --engine-install-url "https://releases.rancher.com/install-docker/19.03.9.sh" \
    boomi-ci-cd

OUTPUT:

Docker Machine Version:  0.16.0, build 702c267f
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:58169
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(flag-lookup) Calling .GetMachineName
(flag-lookup) Calling .DriverName
(flag-lookup) Calling .GetCreateFlags
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:58175
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(boomi-ci-cd) Calling .GetMachineName
(boomi-ci-cd) Calling .DriverName
(boomi-ci-cd) Calling .GetCreateFlags
(boomi-ci-cd) Calling .SetConfigFromFlags
Reading certificate data from /Users/sebastianvaldez/.docker/machine/certs/ca.pem
Decoding PEM data...
Parsing certificate...
Reading certificate data from /Users/sebastianvaldez/.docker/machine/certs/cert.pem
Decoding PEM data...
Parsing certificate...
Running pre-create checks...
(boomi-ci-cd) Calling .PreCreateCheck
(boomi-ci-cd) Calling .GetConfigRaw
Creating machine...
(boomi-ci-cd) Calling .Create
(boomi-ci-cd) Creating SSH key...
(boomi-ci-cd) Creating Digital Ocean droplet...
(boomi-ci-cd) Waiting for IP address to be assigned to the Droplet...
(boomi-ci-cd) DBG | Created droplet ID 256544027, IP address 147.182.201.202
(boomi-ci-cd) Calling .GetConfigRaw
(boomi-ci-cd) Calling .DriverName
(boomi-ci-cd) Calling .DriverName
Waiting for machine to be running, this may take a few minutes...
(boomi-ci-cd) Calling .GetState
(boomi-ci-cd) Calling .GetState
(boomi-ci-cd) Calling .GetState
(boomi-ci-cd) Calling .GetState
(boomi-ci-cd) Calling .GetState
(boomi-ci-cd) Calling .GetState
(boomi-ci-cd) Calling .GetState
(boomi-ci-cd) Calling .GetState
Detecting operating system of created instance...
Waiting for SSH to be available...
Getting to WaitForSSH function...
(boomi-ci-cd) Calling .GetSSHHostname
(boomi-ci-cd) Calling .GetSSHPort
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@147.182.201.202 -o IdentitiesOnly=yes -i /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: exit status 255:
Error getting ssh command 'exit 0' : ssh command error:
command : exit 0
err     : exit status 255
output  :
Getting to WaitForSSH function...
(boomi-ci-cd) Calling .GetSSHHostname
(boomi-ci-cd) Calling .GetSSHPort
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@147.182.201.202 -o IdentitiesOnly=yes -i /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: exit status 255:
Error getting ssh command 'exit 0' : ssh command error:
command : exit 0
err     : exit status 255
output  :
Getting to WaitForSSH function...
(boomi-ci-cd) Calling .GetSSHHostname
(boomi-ci-cd) Calling .GetSSHPort
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@147.182.201.202 -o IdentitiesOnly=yes -i /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: exit status 255:
Error getting ssh command 'exit 0' : ssh command error:
command : exit 0
err     : exit status 255
output  :
Getting to WaitForSSH function...
(boomi-ci-cd) Calling .GetSSHHostname
(boomi-ci-cd) Calling .GetSSHPort
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@147.182.201.202 -o IdentitiesOnly=yes -i /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: <nil>:
Detecting the provisioner...
(boomi-ci-cd) Calling .GetSSHHostname
(boomi-ci-cd) Calling .GetSSHPort
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@147.182.201.202 -o IdentitiesOnly=yes -i /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
cat /etc/os-release
SSH cmd err, output: <nil>: NAME="Ubuntu"
VERSION="20.10 (Groovy Gorilla)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.10"
VERSION_ID="20.10"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=groovy
UBUNTU_CODENAME=groovy

Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
Couldn't set key VERSION_CODENAME, no corresponding struct field found
Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
found compatible host: ubuntu
Provisioning with ubuntu(systemd)...
No storagedriver specified, using overlay2

setting hostname
(boomi-ci-cd) Calling .GetMachineName
(boomi-ci-cd) Calling .GetSSHHostname
(boomi-ci-cd) Calling .GetSSHPort
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@147.182.201.202 -o IdentitiesOnly=yes -i /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo hostname boomi-ci-cd && echo "boomi-ci-cd" | sudo tee /etc/hostname
SSH cmd err, output: <nil>: boomi-ci-cd

(boomi-ci-cd) Calling .GetSSHHostname
(boomi-ci-cd) Calling .GetSSHPort
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@147.182.201.202 -o IdentitiesOnly=yes -i /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:

        if ! grep -xq '.*\sboomi-ci-cd' /etc/hosts; then
            if grep -xq '127.0.1.1\s.*' /etc/hosts; then
                sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 boomi-ci-cd/g' /etc/hosts;
            else
                echo '127.0.1.1 boomi-ci-cd' | sudo tee -a /etc/hosts;
            fi
        fi
SSH cmd err, output: <nil>:
installing base packages
(boomi-ci-cd) Calling .GetSSHHostname
(boomi-ci-cd) Calling .GetSSHPort
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@147.182.201.202 -o IdentitiesOnly=yes -i /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo apt-get update
SSH cmd err, output: <nil>: Hit:1 http://mirrors.digitalocean.com/ubuntu groovy InRelease
Get:2 http://mirrors.digitalocean.com/ubuntu groovy-updates InRelease [115 kB]
Get:3 http://security.ubuntu.com/ubuntu groovy-security InRelease [110 kB]
Get:4 http://mirrors.digitalocean.com/ubuntu groovy-backports InRelease [101 kB]
Get:5 http://mirrors.digitalocean.com/ubuntu groovy-updates/main amd64 Packages [360 kB]
Get:6 http://mirrors.digitalocean.com/ubuntu groovy-updates/main Translation-en [110 kB]
Get:7 http://mirrors.digitalocean.com/ubuntu groovy-updates/main amd64 c-n-f Metadata [8936 B]
Get:8 http://mirrors.digitalocean.com/ubuntu groovy-updates/restricted amd64 Packages [80.7 kB]
Get:9 http://mirrors.digitalocean.com/ubuntu groovy-updates/restricted Translation-en [12.3 kB]
Get:10 http://security.ubuntu.com/ubuntu groovy-security/main amd64 Packages [229 kB]
Get:11 http://mirrors.digitalocean.com/ubuntu groovy-updates/restricted amd64 c-n-f Metadata [448 B]
Get:12 http://mirrors.digitalocean.com/ubuntu groovy-updates/universe amd64 Packages [453 kB]
Get:13 http://mirrors.digitalocean.com/ubuntu groovy-updates/universe Translation-en [101 kB]
Get:14 http://mirrors.digitalocean.com/ubuntu groovy-updates/universe amd64 c-n-f Metadata [9508 B]
Get:15 http://mirrors.digitalocean.com/ubuntu groovy-updates/multiverse amd64 Packages [15.8 kB]
Get:16 http://mirrors.digitalocean.com/ubuntu groovy-updates/multiverse Translation-en [4456 B]
Get:17 http://mirrors.digitalocean.com/ubuntu groovy-updates/multiverse amd64 c-n-f Metadata [624 B]
Get:18 http://security.ubuntu.com/ubuntu groovy-security/main Translation-en [70.8 kB]
Get:19 http://security.ubuntu.com/ubuntu groovy-security/main amd64 c-n-f Metadata [5528 B]
Get:20 http://security.ubuntu.com/ubuntu groovy-security/restricted amd64 Packages [73.8 kB]
Get:21 http://security.ubuntu.com/ubuntu groovy-security/restricted Translation-en [10.8 kB]
Get:22 http://security.ubuntu.com/ubuntu groovy-security/restricted amd64 c-n-f Metadata [404 B]
Get:23 http://security.ubuntu.com/ubuntu groovy-security/universe amd64 Packages [350 kB]
Get:24 http://mirrors.digitalocean.com/ubuntu groovy-backports/main amd64 Packages [2584 B]
Get:25 http://mirrors.digitalocean.com/ubuntu groovy-backports/main Translation-en [1120 B]
Get:26 http://security.ubuntu.com/ubuntu groovy-security/universe Translation-en [66.7 kB]
Get:27 http://security.ubuntu.com/ubuntu groovy-security/universe amd64 c-n-f Metadata [7172 B]
Get:28 http://security.ubuntu.com/ubuntu groovy-security/multiverse amd64 Packages [9744 B]
Get:29 http://security.ubuntu.com/ubuntu groovy-security/multiverse Translation-en [2660 B]
Get:30 http://security.ubuntu.com/ubuntu groovy-security/multiverse amd64 c-n-f Metadata [388 B]
Get:31 http://mirrors.digitalocean.com/ubuntu groovy-backports/main amd64 c-n-f Metadata [400 B]
Get:32 http://mirrors.digitalocean.com/ubuntu groovy-backports/universe amd64 Packages [5704 B]
Get:33 http://mirrors.digitalocean.com/ubuntu groovy-backports/universe Translation-en [1940 B]
Get:34 http://mirrors.digitalocean.com/ubuntu groovy-backports/universe amd64 c-n-f Metadata [256 B]
Fetched 2321 kB in 1s (2056 kB/s)
Reading package lists...

package: action=install name=curl
(boomi-ci-cd) Calling .GetSSHHostname
(boomi-ci-cd) Calling .GetSSHPort
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@147.182.201.202 -o IdentitiesOnly=yes -i /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
DEBIAN_FRONTEND=noninteractive sudo -E apt-get install -y  curl
SSH cmd err, output: <nil>: Reading package lists...
Building dependency tree...
Reading state information...
curl is already the newest version (7.68.0-1ubuntu4.3).
curl set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 28 not upgraded.

Installing Docker...
(boomi-ci-cd) Calling .GetSSHHostname
(boomi-ci-cd) Calling .GetSSHPort
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHKeyPath
(boomi-ci-cd) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@147.182.201.202 -o IdentitiesOnly=yes -i /Users/sebastianvaldez/.docker/machine/machines/boomi-ci-cd/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
if ! type docker; then curl -sSL https://releases.rancher.com/install-docker/19.03.9.sh | sh -; fi
SSH cmd err, output: exit status 100: bash: line 0: type: docker: not found
+ sh -c apt-get update
Hit:1 http://mirrors.digitalocean.com/ubuntu groovy InRelease
Hit:2 http://mirrors.digitalocean.com/ubuntu groovy-updates InRelease
Hit:3 http://mirrors.digitalocean.com/ubuntu groovy-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu groovy-security InRelease
Reading package lists...
+ sh -c apt-get install -y -q apt-transport-https ca-certificates curl software-properties-common
Reading package lists...
Building dependency tree...
Reading state information...
ca-certificates is already the newest version (20210119~20.10.1).
ca-certificates set to manually installed.
curl is already the newest version (7.68.0-1ubuntu4.3).
software-properties-common is already the newest version (0.99.3.1).
software-properties-common set to manually installed.
The following NEW packages will be installed:
  apt-transport-https
0 upgraded, 1 newly installed, 0 to remove and 28 not upgraded.
Need to get 1708 B of archives.
After this operation, 164 kB of additional disk space will be used.
Get:1 http://mirrors.digitalocean.com/ubuntu groovy-updates/universe amd64 apt-transport-https all 2.1.10ubuntu0.3 [1708 B]
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin:
Fetched 1708 B in 0s (32.6 kB/s)
Selecting previously unselected package apt-transport-https.
(Reading database ... 64775 files and directories currently installed.)
Preparing to unpack .../apt-transport-https_2.1.10ubuntu0.3_all.deb ...
Unpacking apt-transport-https (2.1.10ubuntu0.3) ...
Setting up apt-transport-https (2.1.10ubuntu0.3) ...
+ curl -fsSl https://download.docker.com/linux/ubuntu/gpg
+ sh -c apt-key add -
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
+ sh -c add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu groovy stable"
Hit:1 http://mirrors.digitalocean.com/ubuntu groovy InRelease
Get:2 https://download.docker.com/linux/ubuntu groovy InRelease [43.3 kB]
Hit:3 http://mirrors.digitalocean.com/ubuntu groovy-updates InRelease
Hit:4 http://mirrors.digitalocean.com/ubuntu groovy-backports InRelease
Hit:5 http://security.ubuntu.com/ubuntu groovy-security InRelease
Get:6 https://download.docker.com/linux/ubuntu groovy/stable amd64 Packages [7487 B]
Fetched 50.8 kB in 1s (82.5 kB/s)
Reading package lists...
Repository: 'deb [arch=amd64] https://download.docker.com/linux/ubuntu groovy stable'
Description:
Archive for codename: groovy components: stable
More info: https://download.docker.com/linux/ubuntu
Adding repository.
Adding deb entry to /etc/apt/sources.list.d/archive_uri-https_download_docker_com_linux_ubuntu-groovy.list
Adding disabled deb-src entry to /etc/apt/sources.list.d/archive_uri-https_download_docker_com_linux_ubuntu-groovy.list
+ [ ubuntu = debian ]
+ sh -c apt-get update
Hit:1 https://download.docker.com/linux/ubuntu groovy InRelease
Hit:2 http://mirrors.digitalocean.com/ubuntu groovy InRelease
Hit:3 http://mirrors.digitalocean.com/ubuntu groovy-updates InRelease
Hit:4 http://mirrors.digitalocean.com/ubuntu groovy-backports InRelease
Hit:5 http://security.ubuntu.com/ubuntu groovy-security InRelease
Reading package lists...
+ apt-cache madison docker-ce+ head -n 1

+ cut -d   -f 4
+ grep 19.03.9
+ pkg_version=
+ sh -c apt-get install -y -q docker-ce= docker-ce-cli=
Reading package lists...
Building dependency tree...
Reading state information...
E: Version '' for 'docker-ce' was not found
E: Version '' for 'docker-ce-cli' was not found

Error creating machine: Error running provisioning: error installing docker:
notifying bugsnag: [Error creating machine: Error running provisioning: error installing docker: ]
cmckni3 commented 2 years ago

Hey there! I've noticed that the latest docker release (20.10.0 as of this writing) breaks with docker-machine's create phase. The following no longer works:

$ docker-machine create --driver digitalocean --digitalocean-access-token REDACTED --engine-storage-driver overlay2 --digitalocean-image ubuntu-18-04-x64 --digitalocean-size s-1vcpu-1gb --digitalocean-region sfo2 dm-create-test
Running pre-create checks...
Creating machine...
(dm-create-test) Creating SSH key...
(dm-create-test) Creating Digital Ocean droplet...
(dm-create-test) Waiting for IP address to be assigned to the Droplet...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with ubuntu(systemd)...
Installing Docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Error creating machine: Error running provisioning: Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded

This has failed for about the last few hours. Seems like pinning to a custom engine URL works, but I imagine most folks will use this with a broken experience across all drivers. I've currently worked around this by setting:

--engine-install-url "https://releases.rancher.com/install-docker/19.03.9.sh"

Same issue with the AWS provider.

I wasn't able to get to workaround working on a GitLab Runner

hectorm commented 2 years ago

If anyone is interested I've found a workaround for this issue with the Hetzner driver, https://github.com/JonasProgrammer/docker-machine-driver-hetzner/issues/54#issuecomment-899746133.

cmckni3 commented 2 years ago

Oddly enough, started working for me.

I updated Docker and GitLab Docker Machine in my AMI.