Upgrade is broken

[nathanleclaire]

These lines should probably be removed: https://github.com/docker/machine/blob/master/libmachine/provision/ubuntu_systemd.go#L76-L93 as they are causing upgrade to fail due to the removed sources.list file.

[brettdh]

Duplicate of #3009 ?

Or rather, the root cause of it?

[nathanleclaire]

Seems to be so, yes.

[nathanleclaire]

OK, so, anyone who wants a fast fix to this issue, here is a script / sequence of commands that you can run which will put you back on track.

In the following block, make sure to replace mcn_name, lsb_dist, and dist_version with the correct values for your situation (e.g. change lsb_dist to debian and dist_version to jessie if on Debian Jessie).

The following was written for a machine I have running Ubuntu 15.10.

mcn_name='nathanleclaire-swarm-playground'
apt_url='https://apt.dockerproject.org'
lsb_dist='ubuntu'
dist_version='wily'
arch=$(docker-machine ssh ${mcn_name} dpkg --print-architecture)
repo='main'
docker-machine ssh ${mcn_name} "echo deb [arch=${arch}] ${apt_url}/repo ${lsb_dist}-${dist_version} ${repo} | sudo tee /etc/apt/sources.list.d/docker.list"
docker-machine ssh ${mcn_name} sudo apt-get update
docker-machine ssh ${mcn_name} sudo apt-get install -y docker-engine

I'm not too keen on introducing code directly into Machine itself to fix this issue, since a workaround for upgrade coded directly into the Machine source was what caused the issue in the first place.

I will be submitting a patch to remove the offending bits of code. I'll have to confer with the other maintainers on whether or not we'll do a 0.7.1 release to include the fix that removes this destructive bug. Thanks all for your patience in the meantime.

[nathanleclaire]

https://github.com/docker/machine/pull/3312

[mxab]

ok it works for digital ocean, thanks

[nathanleclaire]

👍

[briceburg]

@nathanleclaire FWIW -- this also effected our generically provisioned debian jessie machines. my work around was to shell in and evaluate the get.docker.com script; and to manually fix the systemd unit file ( https://github.com/docker/machine/issues/3009#issuecomment-190778447 ). It's not limited to ubuntu.

[nathanleclaire]

@briceburg Yep, thanks -- the fix above should work for Jessie if the variables are changed as specified.

[AstraSerg]

You can simple do: docker-machine ssh <machine name> sudo mv /etc/apt/sources.list.d/docker.list /etc/apt/sources.list.d/docker-engine.list and everything will be fine.

[Hokutosei]

@nathanleclaire thank you, I tried your script but it seems to be not working for me or did I missed something?, sorry for the late reply. I pasted the script in a small utils_script

upgrade_machine () {
    mcn_name=${mcn_name:-$H}
    apt_url='https://apt.dockerproject.org'
    lsb_dist='ubuntu'
    dist_version='wily'
    arch=$(docker-machine ssh ${mcn_name} dpkg --print-architecture)
    repo='main'
    docker-machine ssh ${mcn_name} "echo deb [arch=${arch}] ${apt_url}/repo ${lsb_dist}-${dist_version} ${repo} | sudo tee /etc/apt/sources.list.d/docker.list"
    docker-machine ssh ${mcn_name} sudo apt-get update
    docker-machine ssh ${mcn_name} sudo apt-get install -y docker-engine    
}

+ source /Users/jeanepaul/Developer/docker/docker-machine/utils_scripts.sh
++ set -e
+ upgrade_machine
+ mcn_name=dev02
+ apt_url=https://apt.dockerproject.org
+ lsb_dist=ubuntu
+ dist_version=wily
++ docker-machine ssh dev02 dpkg --print-architecture
+ arch=amd64
+ repo=main
+ docker-machine ssh dev02 'echo deb [arch=amd64] https://apt.dockerproject.org/repo ubuntu-wily main | sudo tee /etc/apt/sources.list.d/docker.list'
deb [arch=amd64] https://apt.dockerproject.org/repo ubuntu-wily main
+ docker-machine ssh dev02 sudo apt-get update
Ign http://asia-east1.gce.archive.ubuntu.com trusty InRelease
Hit http://asia-east1.gce.archive.ubuntu.com trusty-updates InRelease
Hit http://asia-east1.gce.archive.ubuntu.com trusty Release.gpg
Hit http://asia-east1.gce.archive.ubuntu.com trusty Release
Hit http://asia-east1.gce.archive.ubuntu.com trusty-updates/main Sources
Hit http://asia-east1.gce.archive.ubuntu.com trusty-updates/universe Sources
Hit http://asia-east1.gce.archive.ubuntu.com trusty-updates/main amd64 Packages
Hit http://asia-east1.gce.archive.ubuntu.com trusty-updates/universe amd64 Packages
Hit http://asia-east1.gce.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://asia-east1.gce.archive.ubuntu.com trusty-updates/universe Translation-en
Hit http://asia-east1.gce.archive.ubuntu.com trusty/main Sources
Hit http://asia-east1.gce.archive.ubuntu.com trusty/universe Sources
Hit http://asia-east1.gce.archive.ubuntu.com trusty/main amd64 Packages
Hit http://asia-east1.gce.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://asia-east1.gce.archive.ubuntu.com trusty/main Translation-en
Hit http://asia-east1.gce.archive.ubuntu.com trusty/universe Translation-en
Ign http://asia-east1.gce.archive.ubuntu.com trusty/main Translation-en_US
Ign http://asia-east1.gce.archive.ubuntu.com trusty/universe Translation-en_US
Get:1 https://apt.dockerproject.org ubuntu-wily InRelease
Ign https://apt.dockerproject.org ubuntu-wily InRelease
Hit https://apt.dockerproject.org ubuntu-wily Release.gpg
Hit https://apt.dockerproject.org ubuntu-wily Release
Hit https://apt.dockerproject.org ubuntu-wily/main amd64 Packages
Ign http://archive.canonical.com trusty InRelease
Get:2 https://apt.dockerproject.org ubuntu-wily/main Translation-en_US
Hit http://security.ubuntu.com trusty-security InRelease
Ign https://apt.dockerproject.org ubuntu-wily/main Translation-en_US
Hit http://archive.canonical.com trusty Release.gpg
Ign https://apt.dockerproject.org ubuntu-wily/main Translation-en
Hit http://security.ubuntu.com trusty-security/main Sources
Hit http://archive.canonical.com trusty Release
Hit http://security.ubuntu.com trusty-security/universe Sources
Hit http://archive.canonical.com trusty/partner amd64 Packages
Hit http://security.ubuntu.com trusty-security/main amd64 Packages
Hit http://archive.canonical.com trusty/partner Translation-en
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Reading package lists...
+ docker-machine ssh dev02 sudo apt-get install -y docker-engine
Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  linux-headers-3.16.0-49 linux-headers-3.16.0-49-generic
  linux-headers-3.16.0-67 linux-headers-3.16.0-67-generic
  linux-headers-3.16.0-69 linux-headers-3.16.0-69-generic
  linux-image-3.16.0-49-generic linux-image-3.16.0-67-generic
  linux-image-3.16.0-69-generic linux-image-3.19.0-56-generic
  linux-image-extra-3.16.0-49-generic linux-image-extra-3.16.0-67-generic
  linux-image-extra-3.16.0-69-generic linux-image-extra-3.19.0-56-generic
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  docker-engine
Recommended packages:
  yubico-piv-tool
0 upgraded, 0 newly installed, 0 to remove and 86 not upgraded.
➜  azure git:(master) ✗

this is how I have created the machine btw

docker-machine -D create \
    --driver google \
    --google-project $PROJECT_ID \
    --google-zone $MACHINE_ZONE \
    --google-machine-type $MACHINE_TYPE \
    --google-disk-size $MACHINE_DISK_SIZE $NAME

[nathanleclaire]

@Hokutosei Did you read this bit?

In the following block, make sure to replace mcn_name, lsb_dist, and dist_version with the correct values for your situation (e.g. change lsb_dist to debian and dist_version to jessie if on Debian Jessie).

It looks like your version of Ubuntu may be trusty, not wily. Did you set dist_version=trusty?

[Hokutosei]

@nathanleclaire damn I did not set that! it worked! really sorry for the trouble. script work perfectly thanks you!

[ctroncoso]

this should make it general Just save the following to a file (i.e.: fixit.sh). Then execute it with the name of your machine as a parameter. i.e: $ sh fixit.sh my_machine

Based on @nathanleclaire code.

Modifications:

• made the machine name a parameter.
• distribution name is captured from lsb_release output.
• dist_version is captured from lsb_release output.

dm_name=$1
apt_url='https://apt.dockerproject.org'
lsb_dist=$(docker-machine ssh ${dm_name} "lsb_release -a 2>/dev/null | grep Distributor | cut -f2 | tr '[:upper:]' '[:lower:]'")

dist_version=$(docker-machine ssh ${dm_name} "lsb_release -a 2>/dev/null | grep Codename | cut -f2")
arch=$(docker-machine ssh ${dm_name} dpkg --print-architecture)
repo='main'

docker-machine ssh ${dm_name} "echo deb [arch=${arch}] ${apt_url}/repo ${lsb_dist}-${dist_version} ${repo} | sudo tee /etc/apt/sources.list.d/docker.list"
docker-machine ssh ${dm_name} sudo apt-get update
docker-machine ssh ${dm_name} sudo apt-get install -y docker-engine

Edit: pasted extra debug lines by mistake. Fixed.

[nathanleclaire]

@ctroncoso Script looks pretty good, but are the echo and exit lines in the middle intentional?

[ctroncoso]

Oooops!... @nathanleclaire. Pasted the debug version. Fixed the post.

[s17t]

Experience same issue with softlayer machines. The @ctroncoso's script fixed.

[christianbundy]

Just a quick warning, the above worked for me, but upon running docker-compose up -d I got:

ERROR: for db  Unable to remove filesystem for 985ac64bb671214d61335a609a3605f254b52ca823c47c37002bb96242027a20: remove /var/lib/docker/containers/985ac64bb671214d61335a609a3605f254b52ca823c47c37002bb96242027a20/shm: device or resource busy
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 63, in main
AttributeError: 'ProjectError' object has no attribute 'msg'
docker-compose returned -1

Restarting the machine seems to fix it.

[RobMaskell]

Found my way here because I'm trying to upgrade 1.10.3 to 1.11.1b but the docker-machine is centos image running on GCE so script above won't work and upgrade fails

[nathanleclaire]

@RobMaskell docker-machine upgrade does not work? What is the output for docker-machine --debug upgrade machinename ?

[nathanleclaire]

The referenced bug seems to be Ubuntu/CentOS specific, so it may not be the same issue.

However, it's possible that the upstream packages / install method changed for RH-based distros.

[RobMaskell]

@nathanleclaire had an issue for ages on 1.10.3 that docker on the GCE image would't start up correctly (suspected timing issue) and as it was annoying me I figured it might be fixed in later version so installed the Docker toolbox onto my mac, which upgraded my default virtualbox instance fine. Ran the upgrade on the first of my GCE docker machines and it failed. Looked at a few issues on here and they all seemed to link back to this one. I can't provide the output you ask as I deleted the whole machine and recreated it at 1.11.1b. Good news is it seems to startup ok now.

[Hokutosei]

Greetings,

upgraded my local docker to docker mac, and then upgrade docker-machines, with docker-machine upgrade

➜  accounting git:(develop) dm upgrade dev02
Waiting for SSH to be available...
Detecting the provisioner...
Upgrading docker...
Restarting docker...
➜  accounting git:(develop) dm ls
NAME      ACTIVE   DRIVER       STATE     URL                          SWARM   DOCKER    ERRORS
azure01   -        azure        Running   tcp://xxx.xxx.xx.xxx:2376            v1.11.2
dev       -        virtualbox   Running   tcp://xxx.xxx.xx.xxx:2376            v1.11.0
dev01     -        google                                                      Unknown
dev02     -        google       Running   tcp://xxx.xxx.xx.xxx:2376            v1.11.2

tried the above script again,

    mcn_name=${mcn_name:-$H}
    apt_url='https://apt.dockerproject.org'
    lsb_dist='ubuntu'
    dist_version='trusty'
    arch=$(docker-machine ssh ${mcn_name} dpkg --print-architecture)
    repo='main'
    docker-machine ssh ${mcn_name} "echo deb [arch=${arch}] ${apt_url}/repo ${lsb_dist}-${dist_version} ${repo} | sudo tee /etc/apt/sources.list.d/docker.list"
    docker-machine ssh ${mcn_name} sudo apt-get update
    docker-machine ssh ${mcn_name} sudo apt-get install -y docker-engine

unfortunately, no, not working..

➜  accounting git:(develop) docker version
Client:
 Version:      1.12.0-rc2
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   906eacd
 Built:        Fri Jun 17 20:35:33 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.0-rc2
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   a7119de
 Built:        Fri Jun 17 22:09:20 2016
 OS/Arch:      linux/amd64
 Experimental: true

➜  accounting git:(develop) docker-machine version
docker-machine version 0.7.0, build a650a40

can't even do docker-machine ls without doing unset DOCKER_TLS_VERIFY. with set, It give me Are you trying to connect to a TLS-enabled daemon without TLS? might do this on different post

[christianbundy]

@Hokutosei What's the output of this script? I'd also make sure that you're using Bash, on the off chance there are incompatibilities.

dm_name='dev02'
apt_url='https://apt.dockerproject.org'
lsb_dist=$(docker-machine ssh ${dm_name} "lsb_release -a 2>/dev/null | grep Distributor | cut -f2 | tr '[:upper:]' '[:lower:]'")

dist_version=$(docker-machine ssh ${dm_name} "lsb_release -a 2>/dev/null | grep Codename | cut -f2")
arch=$(docker-machine ssh ${dm_name} dpkg --print-architecture)
repo='main'

docker-machine ssh ${dm_name} "echo deb [arch=${arch}] ${apt_url}/repo ${lsb_dist}-${dist_version} ${repo} | sudo tee /etc/apt/sources.list.d/docker.list"
docker-machine ssh ${dm_name} sudo apt-get update
docker-machine ssh ${dm_name} sudo apt-get install -y docker-engine

[Hokutosei]

@christianbundy sorry, I could not retrieve what are the logs, was on different machines. after an hour and half with frustrations and it was 3am, I gave up and manually install the latest version from this post.

crossing fingers, upgrading azure01

[Hokutosei]

@christianbundy thanks for the help..tried your script, but it seems to be not working..

+ dm_name=azure01
+ apt_url=https://apt.dockerproject.org
++ docker-machine ssh azure01 'lsb_release -a 2>/dev/null | grep Distributor | cut -f2 | tr '\''[:upper:]'\'' '\''[:lower:]'\'''
+ lsb_dist=ubuntu
++ docker-machine ssh azure01 'lsb_release -a 2>/dev/null | grep Codename | cut -f2'
+ dist_version=wily
++ docker-machine ssh azure01 dpkg --print-architecture
+ arch=amd64
+ repo=main
+ docker-machine ssh azure01 'echo deb [arch=amd64] https://apt.dockerproject.org/repo ubuntu-wily main | sudo tee /etc/apt/sources.list.d/docker.list'
deb [arch=amd64] https://apt.dockerproject.org/repo ubuntu-wily main
+ docker-machine ssh azure01 sudo apt-get update
Hit https://apt.dockerproject.org ubuntu-wily InRelease
Hit https://apt.dockerproject.org ubuntu-wily/main amd64 Packages
Hit http://azure.archive.ubuntu.com wily InRelease
Get:1 https://apt.dockerproject.org ubuntu-wily/main Translation-en_US [454 B]
Get:2 https://apt.dockerproject.org ubuntu-wily/main Translation-en [454 B]
Get:3 https://apt.dockerproject.org ubuntu-wily/main Translation-en_US [454 B]
Get:4 https://apt.dockerproject.org ubuntu-wily/main Translation-en [454 B]
Get:5 https://apt.dockerproject.org ubuntu-wily/main Translation-en_US [454 B]
Get:6 https://apt.dockerproject.org ubuntu-wily/main Translation-en [454 B]
Get:7 https://apt.dockerproject.org ubuntu-wily/main Translation-en_US [454 B]
Get:8 https://apt.dockerproject.org ubuntu-wily/main Translation-en [454 B]
Get:9 https://apt.dockerproject.org ubuntu-wily/main Translation-en_US [454 B]
Ign https://apt.dockerproject.org ubuntu-wily/main Translation-en_US

.........
Reading package lists...
+ docker-machine ssh azure01 sudo apt-get install -y docker-engine
Reading package lists...
Building dependency tree...
Reading state information...
docker-engine is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 32 not upgraded.

bash-3.2$ docker-machine ssh azure01
docker-user@azure01:~$ sudo -i
root@azure01:~# docker version
Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:54:25 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:54:25 2016
 OS/Arch:      linux/amd64

bash-3.2$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                          SWARM   DOCKER        ERRORS
azure01   -        azure        Running   tcp://xxx.xx.xxx.xxx:2376            v1.11.2

bash-3.2$ docker version
Client:
 Version:      1.12.0-rc2
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   906eacd
 Built:        Fri Jun 17 20:35:33 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.0-rc2
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   a7119de
 Built:        Fri Jun 17 22:09:20 2016
 OS/Arch:      linux/amd64
 Experimental: true

pardon me, am I missing something?

[christianbundy]

@Hokutosei Did you manually install v1.12.0-rc2? Running docker-machine upgrade just installs the most recent stable version of docker-engine from here, and since v1.12.0 hasn't been released it isn't an install candidate.

[Hokutosei]

@christianbundy installed it from here and prior to 1.12.0-rc2 versions, docker-machine upgrade seems to be not working to me at all.

and yeah, not even RC yet, so until then, its just wait and see... can't downgrade too, lots of work that needs to be done today. it just sucks, upgrading my dev machine for everyday job is a bad idea.

[christianbundy]

@Hokutosei What version are you expecting when you run docker-machine upgrade? It looks like you're already on the most stable version. If you want to upgrade azure01 to v1.12.0-rc2 you can do that by running docker-machine ssh azure01 and then running curl -fsSL https://test.docker.com/ | sh. If you want to be a little more risky you can run curl -fsSL https://experimental.docker.com/ | sh for experimental features, but I don't know how deep into the rabbit hole you want to go.

[Hokutosei]

@christianbundy I was expecting it to be the latest, if not atleast don't give me an error, if not. from my dev machine, doing something to docker-machines (e.g azure01)

Error response from daemon: client is newer than server (client API version: 1.24, server API version: 1.23)

anyway, I am manually upgrading just to fix that error, because I need it to work asap.

thank you, btw.

[schmunk42]

@Hokutosei You can also set export DOCKER_API_VERSION=1.23 to work around that.

[Hokutosei]

@schmunk42 sick! tried it on my old dev virtBox(v1.11.0) it worked like magic 😄

➜  docker-machine git:(master) ✗ dm ls
NAME      ACTIVE   DRIVER       STATE     URL                          SWARM   DOCKER        ERRORS
azure01   -        azure        Running   tcp://xxx.xx.xxx.xxx:2376            v1.12.0-rc2
dev       *        virtualbox   Running   tcp://xxx.xx.xxx.xxx:2376            v1.11.0

➜  docker-machine git:(master) ✗ DOCKER_API_VERSION=1.23 HOST=dev do remote ps -a
+ docker -H tcp://192.168.99.100:2376 ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                   PORTS               NAMES
d5ea64cf83d2        b-eee/nginx-lb      "nginx -g 'daemon off"   4 weeks ago         Exited (0) 2 weeks ago                       tiny_hugle
e6eb9fbbd0fa        b-eee/nginx-lb      "nginx"                  4 weeks ago         Exited (0) 4 weeks ago                       suspicious_goldstine

[gittycat]

@schmunk42 's export DOCKER_API_VERSION=1.xx is the simplest and most elegant way to solve this problem.

[uttamanand]

I am trying to deploy a stack but I am getting an issue -: Command -: docker -H ${SWARM_SERVER_NAME} stack deploy --prune -c mgmt-compose.yml build

Error -: docker stack deploy requires API version 1.25, but the Docker daemon API version is

docker version
Client: Version: 17.12.1-ce API version: 1.35 Go version: go1.9.4 Git commit: 7390fc6 Built: Tue Feb 27 22:17:53 2018 OS/Arch: linux/amd64

Server: Engine: Version: 17.12.1-ce API version: 1.35 (minimum version 1.12) Go version: go1.9.4 Git commit: 7390fc6 Built: Tue Feb 27 22:16:25 2018 OS/Arch: linux/amd64 Experimental: true

OS -: Ubuntu 17.10:latest

Any solution or suggestion will be highly appreciable.

Thanks Uttam