Redundant upload of layers with identical content

[mikehaertl]

I thought that docker 1.10 would improve reuse of layers with identical content. But still I see a lot of layers being moved around whenever I build a new image and push it to docker hub.

Here's an example:

I have built an image from my Dockerfile here: https://github.com/codemix/yii2-dockerbase/blob/master/2.0/apache/Dockerfile

It's based on the official php image (FROM php:5.6.18-apache). So I would expect, that it does not upload the layers from that base image again when I push my image to docker hub. But that's not the case.

Here's the history of my local image:

IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
7662f1a56c2b        About an hour ago   /bin/sh -c composer install --prefer-dist --n   36.5 MB             
05ecce30ee70        About an hour ago   /bin/sh -c #(nop) COPY file:6b1b80b4b5558c3c5   37.19 kB            
09cd57002fb2        About an hour ago   /bin/sh -c #(nop) COPY file:ee5b45a5aa5f049fc   856 B               
54d4a4709e08        About an hour ago   /bin/sh -c #(nop) WORKDIR /var/www/html         0 B                 
e7d7c4cd2255        About an hour ago   /bin/sh -c #(nop) COPY file:51091338068d33161   170 B               
f1f446eb9ba1        About an hour ago   /bin/sh -c #(nop) COPY file:ed6df90ddf5c8b584   1.705 kB            
8db45f8640b3        About an hour ago   /bin/sh -c curl -sS https://getcomposer.org/i   32.84 MB            
dc4e66116e79        About an hour ago   /bin/sh -c apt-get update     && apt-get -y i   111.7 MB            
dfa17f6ccd7d        About an hour ago   /bin/sh -c #(nop) ENV PATH=/usr/local/sbin:/u   0 B                 
18e7589039b8        About an hour ago   /bin/sh -c #(nop) MAINTAINER haertl.mike@gmai   0 B                 
4239932f5198        2 days ago          /bin/sh -c #(nop) CMD ["apache2-foreground"]    0 B                 
<missing>           2 days ago          /bin/sh -c #(nop) EXPOSE 80/tcp                 0 B                 
<missing>           2 days ago          /bin/sh -c #(nop) WORKDIR /var/www/html         0 B                 
<missing>           2 days ago          /bin/sh -c #(nop) COPY file:7f47595ed3e349ddf   132 B               
<missing>           2 days ago          /bin/sh -c #(nop) COPY multi:c650816d97628c05   3.665 kB            
<missing>           2 days ago          /bin/sh -c buildDeps="   $PHP_EXTRA_BUILD_DEP   152.7 MB            
<missing>           2 days ago          /bin/sh -c #(nop) ENV PHP_SHA256=54dd9106c346   0 B                 
<missing>           2 days ago          /bin/sh -c #(nop) ENV PHP_FILENAME=php-5.6.18   0 B                 
<missing>           2 days ago          /bin/sh -c #(nop) ENV PHP_VERSION=5.6.18        0 B                 
<missing>           2 days ago          /bin/sh -c set -xe  && for key in $GPG_KEYS;    16.55 kB            
<missing>           2 days ago          /bin/sh -c #(nop) ENV GPG_KEYS=0BD78B5F97500D   0 B                 
<missing>           2 days ago          /bin/sh -c #(nop) ENV PHP_EXTRA_CONFIGURE_ARG   0 B                 
<missing>           2 days ago          /bin/sh -c #(nop) ENV PHP_EXTRA_BUILD_DEPS=ap   0 B                 
<missing>           2 days ago          /bin/sh -c #(nop) COPY file:cd759b8d41f4b210d   1.52 kB             
<missing>           2 days ago          /bin/sh -c mv /etc/apache2/apache2.conf /etc/   7.115 kB            
<missing>           2 days ago          /bin/sh -c a2dismod mpm_event && a2enmod mpm_   68 B                
<missing>           2 days ago          /bin/sh -c rm -rf /var/www/html && mkdir -p /   0 B                 
<missing>           2 days ago          /bin/sh -c apt-get update && apt-get install    7.146 MB            
<missing>           2 days ago          /bin/sh -c mkdir -p $PHP_INI_DIR/conf.d         0 B                 
<missing>           2 days ago          /bin/sh -c #(nop) ENV PHP_INI_DIR=/usr/local/   0 B                 
<missing>           2 days ago          /bin/sh -c apt-get update && apt-get install    177.2 MB            
<missing>           2 days ago          /bin/sh -c apt-get update && apt-get install    18.64 MB            
<missing>           2 days ago          /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B                 
<missing>           2 days ago          /bin/sh -c #(nop) ADD file:6e3677c176d6d774f0   125.1 MB

Now when I push my custom image, I see this:

The push refers to a repository [docker.io/codemix/yii2-base]
0da57a4401db: Pushed 
556097736914: Pushed 
d4ad6e3ef7e3: Pushed 
71a57f73ca80: Pushed 
7634e82a1ba4: Pushed 
306f40cd5cef: Pushed 
b4b543b0386d: Pushing [=====================================>             ] 84.54 MB/111.7 MB
5f70bf18a086: Layer already exists 
9b250b3b68bc: Pushed 
fb45602bd553: Pushed 
586be5f41d47: Pushed 
37a41489a071: Pushed 
bbf5d3940a19: Pushed 
9adc081e067d: Pushed 
7cf6de835f3e: Pushed 
1720b69c5a62: Pushed 
029b127a2890: Pushed 
25522c9b0b7c: Pushed 
5116fec8cf49: Pushing [========================>                          ]  86.2 MB/177.2 MB
b45d511e3568: Pushed 
ddef881aeaff: Pushing [===================>                               ]  48.9 MB/125.1 MB

So it basically pushed all the layers again. Even ddef88 which is not part of my build but seems to belong to the official php image.

I would have expected that with 1.10 it would only push the id of the content addressable diff to docker hub and then see, that this content is already there. So it should not upload it again.

[aaronlehmann]

Hi @mikehaertl,

The new content addressable storage feature in Docker 1.10 relates to how a Docker daemon storages images locally.

There is a different feature in 1.10 called cross-repository push that addresses the use case you're describing. It allows the engine to prove to a registry that it has access to layers in a different repository. This allows it to skip pushing redundant copies of that layer.

This feature requires registry-side support, which is available in the brand new Registry 2.3. Docker Hub isn't running this new version quite yet, but once they are, pushing custom images to Hub won't require repushing base layers.

[dmp42]

@mikehaertl also worth mentioning: for Hub specific questions and updates, I recommend you reach out to the Hub Support Team: support@docker.com