search

docker / machine

Machine management for a container-centric world
https://docs.docker.com/machine/
Apache License 2.0
6.63k stars 1.97k forks source link

unexpected EOF on remove machine #3416

Open olegsmetanin opened 8 years ago

olegsmetanin commented 8 years ago

Incorrectly created machine using command docker-machine create \ --driver=digitalocean \ --digitalocean-access-token=$DO_TOKEN \ --digitalocean-size=512mb \ --digitalocean-region=nyc3 \ --digitalocean-private-networking=true \ --digitalocean-image=ubuntu-15-04-x64 \ --swarm \ --swarm-discovery consul://${kvip}:8500 \ --engine-opt "cluster-store consul://${kvip}:8500" \ --engine-opt "cluster-advertise eth1:2376" \ --engine-label nodetype=front \ docker-swarm-agent-1 (error in --digitalocean-image=ubuntu-15-04-x64, must be --digitalocean-image=ubuntu-15-10-x64) cannot be removed. $ docker-machine rm docker-swarm-agent-1 About to remove docker-swarm-agent-1 Are you sure? (y/n): y Error removing host "docker-swarm-agent-1": unexpected EOF

bamarni commented 8 years ago

When there has been a failure during the creation process, removing with the force option should usually do the trick : docker-machine rm -f machine. Those error messages could probably be improved though.

nathanleclaire commented 8 years ago

+1 to @bamarni : -f is always intended to be the escape hatch. Not sure what's up with that EOF though.

replaid commented 8 years ago

I hit this today after hitting #3394.

nathanleclaire commented 8 years ago

I've gotten this a bit too... will have to look into it

pnull commented 7 years ago

We're facing exactly the same issue with:

docker-machine version 0.8.2, build e18a919

Machines automatically created be gitlab-ci-multi-runner on digitalocean stay in a defective state:

NAME                                            ACTIVE   DRIVER         STATE     URL                         SWARM   DOCKER    ERRORS
runner-05271b50-ci-runner-1480442822-b39db0ca   -        digitalocean   Error                                         Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480442822-ef9148dd   -        digitalocean   Error                                         Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480442832-a80d1878   -        digitalocean   Error                                         Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480442832-de96936a   -        digitalocean   Error                                         Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480442834-80ffc8d4   -        digitalocean   Error                                         Unknown   dropletID is invalid because cannot be less than 1

When i try to delete them i get the following:

About to remove runner-05271b50-ci-runner-1480442822-b39db0ca
Are you sure? (y/n): y
Error removing host "runner-05271b50-ci-runner-1480442822-b39db0ca": unexpected EOF

The strange thing is, that sometimes the machines exist on DO, sometimes not?!

nathanleclaire commented 7 years ago

curious what inspect shows for one of those, e.g., runner-05271b50-ci-runner-1480442822-b39db0ca.... sounds like the config.json may have gotten corrupted.

nathanleclaire commented 7 years ago

My guess on the EOF is, plugin is panic-ing, so anyone hitting this issue, please make sure to post same output with docker-machine --debug so I can see the logs

pnull commented 7 years ago

@nathanleclaire Sure thing... here you go:

root@gitlab-ci-autoscale-runner:~# docker-machine --debug ls
Docker Machine Version:  0.8.2, build e18a919
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:41957
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:36081
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:41716
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:41469
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:41849
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:45806
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:46082
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:35165
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:45839
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:35635
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:33504
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:42940
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:33666
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:34058
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:45106
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
timeout set to 10s
(runner-05271b50-ci-runner-1480633251-11ab463f) Calling .GetURL
(runner-05271b50-ci-runner-1480624459-e0af5dec) Calling .GetURL
(runner-05271b50-ci-runner-1480625027-c6861d0b) Calling .GetURL
(runner-05271b50-ci-runner-1480625029-9d254439) Calling .GetURL
(runner-05271b50-ci-runner-1480625031-214ca979) Calling .GetURL
(runner-05271b50-ci-runner-1480625033-c77e25c0) Calling .GetURL
(runner-05271b50-ci-runner-1480625035-7f797c24) Calling .GetURL
(runner-05271b50-ci-runner-1480625037-e3dbf764) Calling .GetURL
(runner-05271b50-ci-runner-1480625039-92116735) Calling .GetURL
(runner-05271b50-ci-runner-1480625041-35b53752) Calling .GetURL
(runner-05271b50-ci-runner-1480625601-52d66efc) Calling .GetURL
(runner-05271b50-ci-runner-1480625604-eb9d79d7) Calling .GetURL
(runner-05271b50-ci-runner-1480626154-cb8c7f2a) Calling .GetURL
(runner-05271b50-ci-runner-1480626704-f21a76bd) Calling .GetURL
(runner-05271b50-ci-runner-1480626706-a5de8a9d) Calling .GetURL
(runner-05271b50-ci-runner-1480624459-e0af5dec) Calling .GetState
(runner-05271b50-ci-runner-1480625027-c6861d0b) Calling .GetState
(runner-05271b50-ci-runner-1480626704-f21a76bd) Calling .GetState
(runner-05271b50-ci-runner-1480625601-52d66efc) Calling .GetState
(runner-05271b50-ci-runner-1480625604-eb9d79d7) Calling .GetState
(runner-05271b50-ci-runner-1480625039-92116735) Calling .GetState
(runner-05271b50-ci-runner-1480625037-e3dbf764) Calling .GetState
(runner-05271b50-ci-runner-1480625033-c77e25c0) Calling .GetState
(runner-05271b50-ci-runner-1480625031-214ca979) Calling .GetState
(runner-05271b50-ci-runner-1480625029-9d254439) Calling .GetState
(runner-05271b50-ci-runner-1480626706-a5de8a9d) Calling .GetState
(runner-05271b50-ci-runner-1480626154-cb8c7f2a) Calling .GetState
(runner-05271b50-ci-runner-1480626706-a5de8a9d) Calling .DriverName
(runner-05271b50-ci-runner-1480625601-52d66efc) Calling .DriverName
(runner-05271b50-ci-runner-1480625604-eb9d79d7) Calling .DriverName
(runner-05271b50-ci-runner-1480625029-9d254439) Calling .DriverName
(runner-05271b50-ci-runner-1480625033-c77e25c0) Calling .DriverName
(runner-05271b50-ci-runner-1480625039-92116735) Calling .DriverName
(runner-05271b50-ci-runner-1480625027-c6861d0b) Calling .DriverName
(runner-05271b50-ci-runner-1480626704-f21a76bd) Calling .DriverName
(runner-05271b50-ci-runner-1480625035-7f797c24) Calling .GetState
(runner-05271b50-ci-runner-1480625041-35b53752) Calling .GetState
(runner-05271b50-ci-runner-1480626154-cb8c7f2a) Calling .DriverName
(runner-05271b50-ci-runner-1480624459-e0af5dec) Calling .DriverName
(runner-05271b50-ci-runner-1480625041-35b53752) Calling .DriverName
(runner-05271b50-ci-runner-1480625035-7f797c24) Calling .DriverName
(runner-05271b50-ci-runner-1480625031-214ca979) Calling .DriverName
(runner-05271b50-ci-runner-1480625037-e3dbf764) Calling .DriverName
Reading CA certificate from /root/.docker/machine/certs/ca.pem
Reading client certificate from /root/.docker/machine/certs/cert.pem
Reading client key from /root/.docker/machine/certs/key.pem
(runner-05271b50-ci-runner-1480633251-11ab463f) Calling .DriverName
NAME                                            ACTIVE   DRIVER         STATE     URL                        SWARM   DOCKER    ERRORS
runner-05271b50-ci-runner-1480624459-e0af5dec   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625027-c6861d0b   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625029-9d254439   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625031-214ca979   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625033-c77e25c0   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625035-7f797c24   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625037-e3dbf764   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625039-92116735   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625041-35b53752   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625601-52d66efc   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625604-eb9d79d7   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480626154-cb8c7f2a   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480626704-f21a76bd   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480626706-a5de8a9d   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480633251-11ab463f   -        digitalocean   Running   tcp://138.68.86.154:2376           v1.11.2
Making call to close driver server
(runner-05271b50-ci-runner-1480624459-e0af5dec) Calling .Close
(runner-05271b50-ci-runner-1480624459-e0af5dec) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625027-c6861d0b) Calling .Close
(runner-05271b50-ci-runner-1480625027-c6861d0b) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625029-9d254439) Calling .Close
(runner-05271b50-ci-runner-1480625029-9d254439) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625031-214ca979) Calling .Close
(runner-05271b50-ci-runner-1480625031-214ca979) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625033-c77e25c0) Calling .Close
(runner-05271b50-ci-runner-1480625033-c77e25c0) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625035-7f797c24) Calling .Close
(runner-05271b50-ci-runner-1480625035-7f797c24) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625037-e3dbf764) Calling .Close
(runner-05271b50-ci-runner-1480625037-e3dbf764) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625039-92116735) Calling .Close
(runner-05271b50-ci-runner-1480625039-92116735) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625041-35b53752) Calling .Close
(runner-05271b50-ci-runner-1480625041-35b53752) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625601-52d66efc) Calling .Close
(runner-05271b50-ci-runner-1480625601-52d66efc) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625604-eb9d79d7) Calling .Close
(runner-05271b50-ci-runner-1480625604-eb9d79d7) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480626154-cb8c7f2a) Calling .Close
(runner-05271b50-ci-runner-1480626154-cb8c7f2a) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480626704-f21a76bd) Calling .Close
(runner-05271b50-ci-runner-1480626704-f21a76bd) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480626706-a5de8a9d) Calling .Close
(runner-05271b50-ci-runner-1480626706-a5de8a9d) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480633251-11ab463f) Calling .Close
(runner-05271b50-ci-runner-1480633251-11ab463f) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary

As you can see, this is happening quite often... All these have been "created" within two days.

pnull commented 7 years ago

Here's a example config.json for one of those corrupted machines:

{
    "ConfigVersion": 3,
    "Driver": {
        "IPAddress": "",
        "MachineName": "runner-05271b50-ci-runner-1480625033-c77e25c0",
        "SSHUser": "core",
        "SSHPort": 22,
        "SSHKeyPath": "",
        "StorePath": "/root/.docker/machine",
        "SwarmMaster": false,
        "SwarmHost": "tcp://0.0.0.0:3376",
        "SwarmDiscovery": "",
        "AccessToken": "REMOVED;)",
        "DropletID": 0,
        "DropletName": "",
        "Image": "coreos-stable",
        "Region": "fra1",
        "SSHKeyID": 0,
        "SSHKeyFingerprint": "",
        "Size": "4gb",
        "IPv6": false,
        "Backups": false,
        "PrivateNetworking": true,
        "UserDataFile": ""
    },
    "DriverName": "digitalocean",
    "HostOptions": {
        "Driver": "",
        "Memory": 0,
        "Disk": 0,
        "EngineOptions": {
            "ArbitraryFlags": [],
            "Dns": null,
            "GraphDir": "",
            "Env": [],
            "Ipv6": false,
            "InsecureRegistry": [],
            "Labels": [],
            "LogLevel": "",
            "StorageDriver": "",
            "SelinuxEnabled": false,
            "TlsVerify": true,
            "RegistryMirror": [
                "REMOVED;)"
            ],
            "InstallURL": "https://get.docker.com"
        },
        "SwarmOptions": {
            "IsSwarm": false,
            "Address": "",
            "Discovery": "",
            "Agent": false,
            "Master": false,
            "Host": "tcp://0.0.0.0:3376",
            "Image": "swarm:latest",
            "Strategy": "spread",
            "Heartbeat": 0,
            "Overcommit": 0,
            "ArbitraryFlags": [],
            "ArbitraryJoinFlags": [],
            "Env": null,
            "IsExperimental": false
        },
        "AuthOptions": {
            "CertDir": "/root/.docker/machine/certs",
            "CaCertPath": "/root/.docker/machine/certs/ca.pem",
            "CaPrivateKeyPath": "/root/.docker/machine/certs/ca-key.pem",
            "CaCertRemotePath": "",
            "ServerCertPath": "/root/.docker/machine/machines/runner-05271b50-ci-runner-1480625033-c77e25c0/server.pem",
            "ServerKeyPath": "/root/.docker/machine/machines/runner-05271b50-ci-runner-1480625033-c77e25c0/server-key.pem",
            "ClientKeyPath": "/root/.docker/machine/certs/key.pem",
            "ServerCertRemotePath": "",
            "ServerKeyRemotePath": "",
            "ClientCertPath": "/root/.docker/machine/certs/cert.pem",
            "ServerCertSANs": [],
            "StorePath": "/root/.docker/machine/machines/runner-05271b50-ci-runner-1480625033-c77e25c0"
        }
    },
    "Name": "runner-05271b50-ci-runner-1480625033-c77e25c0"
}
nathanleclaire commented 7 years ago

Yeah, this is wrong:

        "DropletID": 0,

Not sure how it got that way (maybe driver panic) but the config.json is corrupted :( However if you fill in the DropletID to the correct value rm should (hopefully) work properly. Or, you could rm -f and clean up the machines yourself manually if they exist.

pnull commented 7 years ago

@nathanleclaire The real problem is not that i'm not able to resolve this once. The problem is, that this problem occurs multiple times per day!

We're running gitlab-ci-multi-runner (https://gitlab.com/gitlab-org/gitlab-ci-multi-runner) with docker+machine runner type. Machines are started and stopped multiple times per day and fill up the machine list.

Do you have any idea how i can debug this thing to find out WHY the problem occurs?

nathanleclaire commented 7 years ago

Do you have any idea how i can debug this thing to find out WHY the problem occurs?

Thanks -- If GitLab keeps --debug logs of the actions it's doing with machines and/or you can dupe where the error starts to occur it would be a start. It will be needed to pinpoint where the panic/issue is likely to be occurring. At some point (possibly even during create) it seems that there is an error causing the machine serialization to not work properly. The --debug output listed above just shows after the serialization failure has already occurred.. Where/when this error starts to show up? Can you create successfully?

andytime commented 7 years ago

Hi, we're seeing the same issue when using vsphere.

Create

docker-machine --debug create \
--driver vmwarevsphere \
--vmwarevsphere-username=removed \
--vmwarevsphere-password=removed \
--vmwarevsphere-vcenter=removed \
--vmwarevsphere-vcenter-port=443 \
--vmwarevsphere-network=removed\
--vmwarevsphere-datastore=removed \
--vmwarevsphere-datacenter=removed \
--vmwarevsphere-hostsystem='removed'  \
runner-test

Docker Machine Version:  0.12.2, build 9371605
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver vmwarevsphere
Plugin server listening at address 127.0.0.1:35607
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(flag-lookup) Calling .GetMachineName
(flag-lookup) Calling .DriverName
(flag-lookup) Calling .GetCreateFlags
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver vmwarevsphere
Plugin server listening at address 127.0.0.1:33651
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(runner-test) Calling .GetMachineName
(runner-test) Calling .DriverName
(runner-test) Calling .GetCreateFlags
(runner-test) Calling .SetConfigFromFlags
Running pre-create checks...
(runner-test) Calling .PreCreateCheck
(runner-test) DBG | Connecting to vSphere for pre-create checks...
(runner-test) Calling .GetConfigRaw
Creating machine...
(runner-test) Calling .Create
(runner-test) Downloading /root/.docker/machine/cache/boot2docker.iso from http://removed/mirrors/boot2docker/v17.06.0-ce/boot2docker.iso...
(runner-test) 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
(runner-test) Generating SSH Keypair...
(runner-test) Creating VM...
(runner-test) Uploading Boot2docker ISO ...
(runner-test) Reconfiguring VM...
(runner-test) Waiting for VMware Tools to come online...
(runner-test) Provisioning certs and ssh keys...
(runner-test) DBG | Creating Tar key bundle...
(runner-test) Calling .GetConfigRaw
(runner-test) Calling .DriverName
(runner-test) Calling .DriverName
Waiting for machine to be running, this may take a few minutes...
(runner-test) Calling .GetState
Detecting operating system of created instance...
Waiting for SSH to be available...
Getting to WaitForSSH function...
(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.6.10 -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: <nil>:
Detecting the provisioner...
(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.6.10 -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
cat /etc/os-release
SSH cmd err, output: <nil>: NAME=Boot2Docker
VERSION=17.06.0-ce
ID=boot2docker
ID_LIKE=tcl
VERSION_ID=17.06.0-ce
PRETTY_NAME="Boot2Docker 17.06.0-ce (TCL 7.2); HEAD : 0672754 - Thu Jun 29 00:06:31 UTC 2017"
ANSI_COLOR="1;34"
HOME_URL="http://boot2docker.io"
SUPPORT_URL="https://github.com/boot2docker/boot2docker"
BUG_REPORT_URL="https://github.com/boot2docker/boot2docker/issues"

found compatible host: boot2docker
Provisioning with boot2docker...
(runner-test) Calling .GetMachineName
(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.6.10 -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo /usr/bin/sethostname runner-test && echo "runner-test" | sudo tee /var/lib/boot2docker/etc/hostname
SSH cmd err, output: <nil>: Setting hostname to runner-test Done.
runner-test

(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.6.10 -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
netstat -tln
SSH cmd err, output: <nil>: Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 :::22                   :::*                    LISTEN
tcp        0      0 :::2376                 :::*                    LISTEN

(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.6.10 -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo mkdir -p /var/lib/boot2docker
SSH cmd err, output: <nil>:
(runner-test) Calling .GetMachineName
(runner-test) Calling .GetIP
Copying certs to the local machine directory...
generating server cert: /root/.docker/machine/machines/runner-test/server.pem ca-key=/root/.docker/machine/certs/ca.pem private-key=/root/.docker/machine/certs/ca-key.pem org=root.runner-test san=[172.16.6.10 localhost]
(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.6.10 -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo /etc/init.d/docker stop
SSH cmd err, output: <nil>:
(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.6.10 -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
if [ ! -z "$(ip link show docker0)" ]; then sudo ip link delete docker0; fi
SSH cmd err, output: <nil>:
Copying certs to the remote machine...
(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.6.10 -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
printf '%s' '-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----
' | sudo tee /var/lib/boot2docker/ca.pem
SSH cmd err, output: <nil>: -----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----

(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.6.10 -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
printf '%s' '-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----
' | sudo tee /var/lib/boot2docker/server.pem
SSH cmd err, output: <nil>: -----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----

(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.6.10 -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
printf '%s' '-----BEGIN RSA PRIVATE KEY-----

-----END RSA PRIVATE KEY-----
' | sudo tee /var/lib/boot2docker/server-key.pem
SSH cmd err, output: <nil>: -----BEGIN RSA PRIVATE KEY-----

-----END RSA PRIVATE KEY-----

(runner-test) Calling .GetURL
(runner-test) Calling .DriverName
Setting Docker configuration on the remote daemon...
(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.6.10 -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo mkdir -p /var/lib/boot2docker && printf %s "
EXTRA_ARGS='
--label provider=vmwarevsphere

'
CACERT=/var/lib/boot2docker/ca.pem
DOCKER_HOST='-H tcp://0.0.0.0:2376'
DOCKER_STORAGE=aufs
DOCKER_TLS=auto
SERVERKEY=/var/lib/boot2docker/server-key.pem
SERVERCERT=/var/lib/boot2docker/server.pem

" | sudo tee /var/lib/boot2docker/profile
SSH cmd err, output: <nil>:
EXTRA_ARGS='
--label provider=vmwarevsphere

'
CACERT=/var/lib/boot2docker/ca.pem
DOCKER_HOST='-H tcp://0.0.0.0:2376'
DOCKER_STORAGE=aufs
DOCKER_TLS=auto
SERVERKEY=/var/lib/boot2docker/server-key.pem
SERVERCERT=/var/lib/boot2docker/server.pem

(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.6.10 -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo /etc/init.d/docker start
SSH cmd err, output: <nil>: Device "eth1" does not exist.
Need TLS certs for runner-test,127.0.0.1,172.16.6.10
-------------------
Generate server cert
/usr/local/bin/generate_cert --host=runner-test,127.0.0.1,172.16.6.10 --ca=/var/lib/boot2docker/ca.pem --ca-key=/var/lib/boot2docker/tls/cakey.pem --cert=/var/lib/boot2docker/server.pem --key=/var/lib/boot2docker/server-key.pem --org=Boot2Docker
2017/08/11 15:47:59 Preventing overwrite: the following files already exist: "/var/lib/boot2docker/server.pem" "/var/lib/boot2docker/server-key.pem". To overwrite files, add `--overwrite`.

(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.6.10 -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
netstat -tln
SSH cmd err, output: <nil>: Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 :::22                   :::*                    LISTEN
tcp        0      0 :::2376                 :::*                    LISTEN

(runner-test) Calling .GetIP
Checking connection to Docker...
(runner-test) Calling .GetURL
Reading CA certificate from /root/.docker/machine/certs/ca.pem
Reading client certificate from /root/.docker/machine/certs/cert.pem
Reading client key from /root/.docker/machine/certs/key.pem
Docker is up and running!
Reticulating splines...
(runner-test) Calling .GetConfigRaw
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env runner-test
Making call to close driver server
(runner-test) Calling .Close
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(flag-lookup) Calling .Close
Successfully made call to close driver server
Making call to close connection to plugin binary

Remove


docker-machine --debug rm runner-test
Docker Machine Version:  0.12.2, build 9371605
About to remove runner-test
WARNING: This action will delete both local reference and remote instance.
Are you sure? (y/n): y
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver vmwarevsphere
Plugin server listening at address 127.0.0.1:34485
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(runner-test) Calling .Remove
(runner-test) DBG | panic: runtime error: invalid memory address or nil pointer dereference
(runner-test) DBG | [signal SIGSEGV: segmentation violation code=0x1 addr=0xa0 pc=0xedd8e6]
(runner-test) DBG |
(runner-test) DBG | goroutine 10 [running]:
(runner-test) DBG | github.com/docker/machine/vendor/github.com/vmware/govmomi/task.(*taskCallback).fn(0xc4202ea400, 0x0, 0x0, 0x0, 0x0)
(runner-test) DBG |     /go/src/github.com/docker/machine/vendor/github.com/vmware/govmomi/task/wait.go:73 +0x296
(runner-test) DBG | github.com/docker/machine/vendor/github.com/vmware/govmomi/task.(*taskCallback).(github.com/docker/machine/vendor/github.com/vmware/govmomi/task.fn)-fm(0x0, 0x0, 0x0, 0x0)
(runner-test) DBG |     /go/src/github.com/docker/machine/vendor/github.com/vmware/govmomi/task/wait.go:120 +0x48
(runner-test) DBG | github.com/docker/machine/vendor/github.com/vmware/govmomi/property.waitLoop(0x1d167c0, 0xc42032afc0, 0xc42011b8c0, 0xc42040b7c8, 0x0, 0x0)
(runner-test) DBG |     /go/src/github.com/docker/machine/vendor/github.com/vmware/govmomi/property/wait.go:140 +0x1b2
(runner-test) DBG | github.com/docker/machine/vendor/github.com/vmware/govmomi/property.Wait(0x1d167c0, 0xc42032afc0, 0xc42040b870, 0xc4202c43f8, 0x4, 0xc4202c44a0, 0xb, 0xc4202c4580, 0x1, 0x1, ...)
(runner-test) DBG |     /go/src/github.com/docker/machine/vendor/github.com/vmware/govmomi/property/wait.go:67 +0x2d7
(runner-test) DBG | github.com/docker/machine/vendor/github.com/vmware/govmomi/task.Wait(0x1d167c0, 0xc42032afc0, 0xc4202c43f8, 0x4, 0xc4202c44a0, 0xb, 0xc42040b870, 0x0, 0x0, 0x0, ...)
(runner-test) DBG |     /go/src/github.com/docker/machine/vendor/github.com/vmware/govmomi/task/wait.go:120 +0x17e
(runner-test) DBG | github.com/docker/machine/vendor/github.com/vmware/govmomi/object.(*Task).WaitForResult(0xc42021a570, 0x1d167c0, 0xc42032afc0, 0x0, 0x0, 0xc42011b260, 0x2e, 0x0)
(runner-test) DBG |     /go/src/github.com/docker/machine/vendor/github.com/vmware/govmomi/object/task.go:51 +0x137
(runner-test) DBG | github.com/docker/machine/vendor/github.com/vmware/govmomi/object.(*Task).Wait(0xc42021a570, 0x1d167c0, 0xc42032afc0, 0xc4202f3130, 0xb)
(runner-test) DBG |     /go/src/github.com/docker/machine/vendor/github.com/vmware/govmomi/object/task.go:45 +0x51
(runner-test) DBG | github.com/docker/machine/drivers/vmwarevsphere.(*Driver).Remove(0xc4202150e0, 0x0, 0x0)
(runner-test) DBG |     /go/src/github.com/docker/machine/drivers/vmwarevsphere/vsphere.go:839 +0x6f2
(runner-test) DBG | github.com/docker/machine/libmachine/drivers/rpc.(*RPCServerDriver).Remove(0xc420121080, 0x1d73f60, 0x1d73f60, 0x0, 0x0)
(runner-test) DBG |     /go/src/github.com/docker/machine/libmachine/drivers/rpc/server_driver.go:205 +0x36
(runner-test) DBG | reflect.Value.call(0xc42012a840, 0xc420122270, 0x13, 0x13dad6e, 0x4, 0xc42003df20, 0x3, 0x3, 0x0, 0x0, ...)
(runner-test) DBG |     /usr/local/go/src/reflect/value.go:434 +0x91f
(runner-test) DBG | reflect.Value.Call(0xc42012a840, 0xc420122270, 0x13, 0xc420028f20, 0x3, 0x3, 0x0, 0x0, 0x0)
(runner-test) DBG |     /usr/local/go/src/reflect/value.go:302 +0xa4
(runner-test) DBG | net/rpc.(*service).call(0xc42011f4c0, 0xc42011e100, 0xc42011d650, 0xc420125a00, 0xc42022c2c0, 0x106e840, 0x1d73f60, 0x16, 0x106e840, 0x1d73f60, ...)
(runner-test) DBG |     /usr/local/go/src/net/rpc/server.go:387 +0x144
(runner-test) DBG | created by net/rpc.(*Server).ServeCodec
(runner-test) DBG |     /usr/local/go/src/net/rpc/server.go:481 +0x404
Error removing host "runner-test": unexpected EOF
maxpain commented 6 years ago

+1

nathanaherne commented 6 years ago

I am getting the same "dropletID is invalid because cannot be less than 1" issue with docker-machine version 0.13.0, build 9ba6da9.

Happy to post further logs if that would be helpful.

cyclux commented 6 years ago

Same here. docker-machine version 0.14.0, build 89b8332 in combination with digitalocean driver.

rnwolf commented 5 years ago

+1

PingusPepan commented 5 years ago

Hey guys, I have created a workaround around this issue: link It requires jq on the management droplet and a Digitalocean read-access API token. I have it in my crontab running every 30 minutes. Hope it helps at least a bit

zeppelinen commented 5 years ago

+1 docker-machine version 0.16.1, build cce350d7

agalisteo commented 5 years ago

+1 Running gitlabi-runner with docker+machine runner type.

awalias commented 4 years ago

+1 docker-machine version 0.16.1, build cce350d7 with digital ocean

klutchell commented 4 years ago

+1

$ docker-machine --version
docker-machine version 0.16.0, build 702c267f
$ gitlab-runner --version
Version:      12.7.1
Git revision: 003fe500
Git branch:   12-7-stable
GO version:   go1.13.5
Built:        2020-01-23T09:08:54+0000
OS/Arch:      linux/amd64
J7mbo commented 4 years ago

Got this as well:

$ docker-machine --version
docker-machine version 0.16.0, build 702c267f

And this is my first time using docker-machine - not exactly a great experience so far...

Latest version MacOS. Latest version docker. Latest version docker-machine.

This happened because the process halted half-way through. I can't remember the exact conditions, but it was something like an image not being available, or the wrong digital ocean image (like ubuntu-10-x61) causing it to fail, and then I had to --force.