Docker 1.10 fails

[hjacobs]

The current release candidate 1.10 of Docker fails to pull images from Pier One:

$ docker pull registry.opensource.zalan.do/stups/zmon-demo-bootstrap:cd20
Error response from daemon: invalid format length
$ dpkg -l | grep docker-engine
ii  docker-engine                         1.10.0~rc2-0~wily                          amd64        Docker: the open-source application container engine

[hjacobs]

Apparently the error is thrown during parsing of the JSON signature: https://github.com/docker/docker/blob/729c9a97822ebee2c978a322d37060454af6bc66/vendor/src/github.com/docker/libtrust/jsonsign.go#L526

Fun fact: the "protected" part is simply base64 encoded JSON with content length and a tail:

http https://registry.opensource.zalan.do/v2/stups/zmon-demo-bootstrap/manifests/cd20
...
"signatures": [
        {
            "header": {
                "alg": "ES256",
                "jwk": {
                    "crv": "P-256",
                    "kid": "TSB7:LZ62:NMUV:BCCA:ZWQS:XBIF:RFSB:5A6Q:JNOE:5YU2:TEAR:XNNV",
                    "kty": "EC",
                    "x": "KIcr3qftcEQ4yVp5QmN0LWstEyvZFfsE_QNXr32g22E",
                    "y": "Sc5TWRJEoQw3iF6SLTdr-q8f7lpFIkwbYY7NQmLs3gw"
                }
            },
            "protected": "eyJmb3JtYXRMZW5ndGgiOjIwMzU0LCJmb3JtYXRUYWlsIjoiQ24wIiwidGltZSI6IjIwMTYtMDEtMzBUMjA6MzM6NTdaIn0",
            "signature": "0NX0w6_O7gFi3nEPJFnHGoajSM0b3AnWzwyWDyulZREtcSV_JsFWZ3wwnu8DzReiXgmF5qCN9bIUww0RC-QqxA"
        }
    ],

$ base64 -d
eyJmb3JtYXRMZW5ndGgiOjIwMzU0LCJmb3JtYXRUYWlsIjoiQ24wIiwidGltZSI6IjIwMTYtMDEtMzBUMjA6MzM6NTdaIn0
{"formatLength":20354,"formatTail":"Cn0","time":"2016-01-30T20:33:57Z"}

And now guess what the tail is? Right...

$ base64 -d
Cn0

}

So apparently we just need to add some whitespace to the response JSON to make Docker happy?

[elgalu]

I can also recreate on docker-engine 1.10.0-0~wily

[ahmed1490]

same bug.. any progress in this issue?

[elgalu]

Until this issue is fixed and released the workaround is to downgrade to 1.9.1:

####################
# Downgrade docker #
####################
sudo service docker stop
sudo mv /usr/bin/docker /usr/bin/docker.1.10
cd /tmp
wget https://get.docker.com/builds/Linux/x86_64/docker-1.9.1
sudo mv docker-1.9.1 /usr/bin/docker
sudo chmod 0755 /usr/bin/docker
sudo service docker start

[ahmed1490]

for those with mac.. running this command worked for me

docker-machine create dev --driver virtualbox --virtualbox-boot2docker-url https://github.com/boot2docker/boot2docker/releases/download/v1.9.1/boot2docker.iso --engine-storage-driver overlay

eval $(docker-machine env dev)

[hjacobs]

FYI: how to downgrade on Ubuntu 15.10 ("wily"):

sudo apt-get install docker-engine=1.9.1-0~wily

[whiskeysierra]

:+1:

[DaKine23]

FYI: how to downgrade on Fedora

sudo dnf downgrade docker

[ovolynets]

Downgrade on arch linux (check your docker version):

sudo pacman -U /var/cache/pacman/pkg/docker-1\:1.9.1-2-x86_64.pkg.tar.xz

This option assumes you did not clear the local pacman cache. Otherwise address to the official documentation from ArchWiki. To add docker to the ignored packages, edit the following line in /etc/pacman.conf:

# Pacman won't upgrade packages listed in IgnorePkg and members of IgnoreGroup
IgnorePkg   = docker

[whiskeysierra]

@hjacobs Would adding the whitespace break old clients?

[hjacobs]

@whiskeysierra it's still valid JSON and old Docker versions do not verify any signature AFAIK

[whiskeysierra]

So in theory this could be fixed and we could start using 1.10?

[hjacobs]

@whiskeysierra yes, of course --- "somebody" needs to invest time, that's all :smirk:

[elgalu]

Is this really "in progress" or just tagged? :P FYI I can also recreate on current latest 1.10.3

[hjacobs]

@elgalu @whiskeysierra wanted to take a look at it.

[gnuton]

Latest ubuntu 16.04 doesn't have 1.9 in its repos. :(

gnuton@hal:~/GIT/escluster$ apt-cache madison docker.io docker.io | 1.10.2-0ubuntu4 | http://fi.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages

so as workaround you must use the 15.10 packages

# echo "deb https://apt.dockerproject.org/repo ubuntu-wily main" >> /etc/apt/sources.list
# apt-get update
# apt-get install docker-engine=1.9.1-0~wily

[outime]

If you've downgraded the VM in OS X but the client is still the latest one, you may get this:

docker: Error response from daemon: client is newer than server (client API version: 1.22, server API version: 1.21).

You can easily set the DOCKER_API_VERSION env variable to the server API version (1.21) to override the one used by the client.

[prayerslayer]

TODO:

• Wait for dakrone/cheshire#100
• Deduplicate some parts of the code (e.g. pretty print options)
• PROFIT!?

[hjacobs]

The fixes seem to work fine (tested with registry.opensource.zalan.do and 1.10.3 Docker client)

[prayerslayer]

docker run registry.opensource.zalan.do/stups/node:4.4-20

Works on Mac 10.10.5 with Docker 1.11