search

moby / buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
https://github.com/moby/moby/issues/34227
Apache License 2.0
8.01k stars 1.12k forks source link

pip wheel bustes cache #1099

Open FernandoMiguel opened 5 years ago

FernandoMiguel commented 5 years ago 
# syntax=docker/dockerfile:experimental

FROM python:3.6-alpine AS ms-python-wheel

WORKDIR /src

RUN --mount=type=cache,id=apk,sharing=locked,target=/var/cache/apk ln -vs /var/cache/apk /etc/apk/cache && \
    apk add --update --virtual build-dependencies \
    gcc \
    build-base \
    libffi-dev \
    libressl-dev \
    postgresql-dev

COPY . /src/
RUN --mount=type=cache,id=wheel,sharing=locked,target=/root \
    pip wheel -e .
/usr/libexec/docker/cli-plugins/docker-buildx bake --progress plain -f .jenkins/docker-compose.images-base.1.yml -f .jenkins/buildx.images-base.1.yml flask-core

.jenkins/buildx.images-base.1.yml

  # Python-flask-core
  flask-core:
    build:
      context: ./.Flask-core
      dockerfile: buildx

.jenkins/docker-compose.images-base.1.yml

  flask-core:
    image: flask-core-wheel:${WHEEL_VERSION:-0.4}-${COMMIT_ID}
    build:
      context: ../.Flask-core
      dockerfile: Dockerfile

this is the 2nd run, after cache has been created from a previous successful build

Building Base Images
/usr/libexec/docker/cli-plugins/docker-buildx bake --progress plain -f .jenkins/docker-compose.images-base.1.yml -f .jenkins/buildx.images-base.1.yml
time="2019-07-28T13:47:28Z" level=error msg="Error parsing the serverURL" error="docker-credential-ecr-login can only be used with Amazon Elastic Container Registry." serverURL="https://index.docker.io/v1/"
#1 [internal] load .dockerignore
#1 transferring context: 50B done
#1 DONE 0.0s

#2 [internal] load build definition from buildx
#2 transferring dockerfile: 449B done
#2 DONE 0.0s

#3 resolve image config for docker.io/docker/dockerfile:experimental
#3 DONE 0.3s

#4 docker-image://docker.io/docker/dockerfile:experimental@sha256:9022e9111...
#4 CACHED

#5 [internal] load metadata for docker.io/library/python:3.6-alpine
time="2019-07-28T13:47:29Z" level=error msg="Error parsing the serverURL" error="docker-credential-ecr-login can only be used with Amazon Elastic Container Registry." serverURL="https://index.docker.io/v1/"
#5 DONE 0.3s

#6 [1/5] FROM docker.io/library/python:3.6-alpine@sha256:6073c91ab2c78220c1...
#6 DONE 0.0s

#9 [internal] load build context
#9 transferring context: 5.54kB done
#9 DONE 0.0s

#7 [2/5] WORKDIR /src
#7 CACHED

#8 [3/5] RUN --mount=type=cache,id=apk,sharing=locked,target=/var/cache/apk...
#8 CACHED

#10 [4/5] COPY . /src/
#10 CACHED

#11 [5/5] RUN --mount=type=cache,id=wheel,sharing=locked,target=/root   pip w...
#11 1.280 Obtaining file:///src
#11 1.712 Collecting amqp==2.4.2 (from Flask-Core==0.4.0)
#11 1.833   Using cached https://files.pythonhosted.org/packages/42/ec/cbbaa8f75be8cbd019afb9d63258e2bdc95242f8c46a54bb90db5fef03bd/amqp-2.4.2-py2.py3-none-any.whl
#11 1.839   Saved ./amqp-2.4.2-py2.py3-none-any.whl
#11 1.841 Collecting amqplib==1.0.2 (from Flask-Core==0.4.0)
#11 1.859   Using cached https://files.pythonhosted.org/packages/75/b7/8c2429bf8d92354a0118614f9a4d15e53bc69ebedce534284111de5a0102/amqplib-1.0.2.tgz
#11 2.034 Collecting anyjson==0.3.3 (from Flask-Core==0.4.0)
#11 2.055   Using cached https://files.pythonhosted.org/packages/c3/4d/d4089e1a3dd25b46bebdb55a992b0797cff657b4477bc32ce28038fdecbc/anyjson-0.3.3.tar.gz
#11 2.244 Collecting asn1crypto==0.24.0 (from Flask-Core==0.4.0)
#11 2.272   Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
#11 2.278   Saved ./asn1crypto-0.24.0-py2.py3-none-any.whl
#11 2.280 Collecting beautifulsoup4==4.7.1 (from Flask-Core==0.4.0)
#11 2.308   Using cached https://files.pythonhosted.org/packages/1d/5d/3260694a59df0ec52f8b4883f5d23b130bc237602a1411fa670eae12351e/beautifulsoup4-4.7.1-py3-none-any.whl
#11 2.314   Saved ./beautifulsoup4-4.7.1-py3-none-any.whl
#11 2.318 Collecting billiard==3.6.0.0 (from Flask-Core==0.4.0)
#11 2.361   Using cached https://files.pythonhosted.org/packages/3c/a0/5a01bed569cf1af8fcdd4294590acb980a2788d1594f4ccd678b6fea8644/billiard-3.6.0.0.tar.gz
#11 2.598 Collecting blinker==1.4 (from Flask-Core==0.4.0)
#11 2.620   Using cached https://files.pythonhosted.org/packages/1b/51/e2a9f3b757eb802f61dc1f2b09c8c99f6eb01cf06416c0671253536517b6/blinker-1.4.tar.gz
#11 2.816 Collecting bugsnag==3.5.2 (from Flask-Core==0.4.0)
#11 2.928   Using cached https://files.pythonhosted.org/packages/4d/fe/e9f4f98950ffc211a9ed773e03240c799a209beb205b4da095bde0c96981/bugsnag-3.5.2-py3-none-any.whl
#11 2.932   Saved ./bugsnag-3.5.2-py3-none-any.whl
#11 2.936 Collecting certifi==2019.3.9 (from Flask-Core==0.4.0)
#11 2.966   Using cached https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl
#11 2.970   Saved ./certifi-2019.3.9-py2.py3-none-any.whl
#11 2.972 Collecting cffi==1.12.3 (from Flask-Core==0.4.0)
#11 3.143   Saved ./cffi-1.12.3-cp36-cp36m-linux_x86_64.whl
#11 3.145 Collecting chardet==3.0.4 (from Flask-Core==0.4.0)
#11 3.166   Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
#11 3.173   Saved ./chardet-3.0.4-py2.py3-none-any.whl
#11 3.175 Collecting Click==7.0 (from Flask-Core==0.4.0)
#11 3.207   Using cached https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl
#11 3.211   Saved ./Click-7.0-py2.py3-none-any.whl
#11 3.213 Collecting configobj==5.0.6 (from Flask-Core==0.4.0)
#11 3.233   Using cached https://files.pythonhosted.org/packages/64/61/079eb60459c44929e684fa7d9e2fdca403f67d64dd9dbac27296be2e0fab/configobj-5.0.6.tar.gz
#11 3.417 Collecting cryptography==2.6.1 (from Flask-Core==0.4.0)
#11 3.666   Using cached https://files.pythonhosted.org/packages/07/ca/bc827c5e55918ad223d59d299fff92f3563476c3b00d0a9157d9c0217449/cryptography-2.6.1.tar.gz
#11 3.739   Installing build dependencies: started
#11 4.989   Installing build dependencies: finished with status 'done'
#11 4.992   Getting requirements to build wheel: started
#11 5.218   Getting requirements to build wheel: finished with status 'done'
#11 5.220     Preparing wheel metadata: started
#11 5.880     Preparing wheel metadata: finished with status 'done'
#11 5.900 Collecting cssmin==0.2.0 (from Flask-Core==0.4.0)
#11 5.995   Using cached https://files.pythonhosted.org/packages/8e/d8/dc9da69bb186303f7ab41adef0a5b6d34da2fdba006827620877760241c3/cssmin-0.2.0.tar.gz
#11 6.176 Collecting cssprefixer==1.4.1 (from Flask-Core==0.4.0)
#11 6.273   Using cached https://files.pythonhosted.org/packages/ab/60/375c68c3b5e7fe0da73041d533c083f7bd02da17ba9c953695496e5f2338/cssprefixer-1.4.1.tar.gz
#11 6.451 Collecting cssutils==1.0.2 (from Flask-Core==0.4.0)
#11 6.501   Using cached https://files.pythonhosted.org/packages/6b/15/a9fb9010f58d1c55dd0b7779db2334feb9a572d407024f39a60f44293861/cssutils-1.0.2-py3-none-any.whl
#11 6.526   Saved ./cssutils-1.0.2-py3-none-any.whl
#11 6.528 Collecting docopt==0.4.0 (from Flask-Core==0.4.0)
#11 6.546   Using cached https://files.pythonhosted.org/packages/40/24/3b4941b5cdfe4d09c522bf4849fd28ddee4abfa4b48223bc3611c325a216/docopt-0.4.0.tar.gz
#11 6.723 Collecting docutils==0.14 (from Flask-Core==0.4.0)
#11 6.748   Using cached https://files.pythonhosted.org/packages/36/fa/08e9e6e0e3cbd1d362c3bbee8d01d0aedb2155c4ac112b19ef3cae8eed8d/docutils-0.14-py3-none-any.whl
#11 6.780   Saved ./docutils-0.14-py3-none-any.whl
#11 6.781 Collecting Flask==1.0.2 (from Flask-Core==0.4.0)
#11 6.809   Using cached https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl
#11 6.814   Saved ./Flask-1.0.2-py2.py3-none-any.whl
#11 6.827 Collecting Flask-Assets==0.12 (from Flask-Core==0.4.0)
#11 6.922   Using cached https://files.pythonhosted.org/packages/86/ff/6000451570745d7a90847f6528d96d6b24c800eaaf9f26cf398accd8cee5/Flask-Assets-0.12.tar.gz
#11 7.111 Collecting flask-mongoengine==0.9.5 (from Flask-Core==0.4.0)
#11 7.136   Using cached https://files.pythonhosted.org/packages/20/53/1bb8ad34ad5c2047a11651290325e55086bc18fce7cfdbbe6f5522bd0ae5/flask-mongoengine-0.9.5.tar.gz
#11 9.794 Collecting Flask-Script==2.0.6 (from Flask-Core==0.4.0)
#11 9.818   Using cached https://files.pythonhosted.org/packages/00/a4/cd587b2b19f043b65bf33ceda2f6e4e6cdbd0ce18d01a52b9559781b1da6/Flask-Script-2.0.6.tar.gz
#11 10.01 Collecting Flask-SQLAlchemy==2.4.0 (from Flask-Core==0.4.0)
#11 10.04   Using cached https://files.pythonhosted.org/packages/08/ca/582442cad71504a1514a2f053006c8bb128844133d6076a4df17117545fa/Flask_SQLAlchemy-2.4.0-py2.py3-none-any.whl
#11 10.04   Saved ./Flask_SQLAlchemy-2.4.0-py2.py3-none-any.whl
#11 10.04 Collecting Flask-WTF==0.14.2 (from Flask-Core==0.4.0)
#11 10.07   Using cached https://files.pythonhosted.org/packages/60/3a/58c629472d10539ae5167dc7c1fecfa95dd7d0b7864623931e3776438a24/Flask_WTF-0.14.2-py2.py3-none-any.whl
#11 10.07   Saved ./Flask_WTF-0.14.2-py2.py3-none-any.whl
#11 10.07 Collecting gevent==1.4.0 (from Flask-Core==0.4.0)
#11 10.27   Using cached https://files.pythonhosted.org/packages/ed/27/6c49b70808f569b66ec7fac2e78f076e9b204db9cf5768740cff3d5a07ae/gevent-1.4.0.tar.gz
#11 11.01 Collecting gevent-socketio-hartwork==0.3.6.post201803091952 (from Flask-Core==0.4.0)
#11 11.11   Using cached https://files.pythonhosted.org/packages/4b/ec/c19e293be0532220f0e7fc030f2e1c92c2e505073de8fd57fa5c05cad880/gevent_socketio_hartwork-0.3.6.post201803091952-py3-none-any.whl
#11 11.11   Saved ./gevent_socketio_hartwork-0.3.6.post201803091952-py3-none-any.whl
#11 11.12 Collecting gevent-websocket==0.10.1 (from Flask-Core==0.4.0)
#11 11.22   Using cached https://files.pythonhosted.org/packages/7b/84/2dc373eb6493e00c884cc11e6c059ec97abae2678d42f06bf780570b0193/gevent_websocket-0.10.1-py3-none-any.whl
#11 11.22   Saved ./gevent_websocket-0.10.1-py3-none-any.whl
#11 11.22 Collecting greenlet==0.4.15 (from Flask-Core==0.4.0)
#11 11.30   Using cached https://files.pythonhosted.org/packages/f8/e8/b30ae23b45f69aa3f024b46064c0ac8e5fcb4f22ace0dca8d6f9c8bbe5e7/greenlet-0.4.15.tar.gz
#11 11.48 Collecting html5lib==1.0.1 (from Flask-Core==0.4.0)
#11 11.51   Using cached https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl
#11 11.51   Saved ./html5lib-1.0.1-py2.py3-none-any.whl
#11 11.52 Collecting idna==2.8 (from Flask-Core==0.4.0)
#11 11.55   Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
#11 11.55   Saved ./idna-2.8-py2.py3-none-any.whl
#11 11.55 Collecting itsdangerous==1.1.0 (from Flask-Core==0.4.0)
#11 11.57   Using cached https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
#11 11.57   Saved ./itsdangerous-1.1.0-py2.py3-none-any.whl
#11 11.58 Collecting Jinja2==2.10.1 (from Flask-Core==0.4.0)
#11 11.60   Using cached https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl
#11 11.61   Saved ./Jinja2-2.10.1-py2.py3-none-any.whl
#11 11.61 Collecting kombu==4.5.0 (from Flask-Core==0.4.0)
#11 11.68   Using cached https://files.pythonhosted.org/packages/b7/af/1914e93314f1b98756d5c5e366193124a0ffaab0e6d0e51e0f6f65fa851d/kombu-4.5.0-py2.py3-none-any.whl
#11 11.69   Saved ./kombu-4.5.0-py2.py3-none-any.whl
#11 11.71 Collecting lockfile==0.12.2 (from Flask-Core==0.4.0)
#11 11.73   Using cached https://files.pythonhosted.org/packages/c8/22/9460e311f340cb62d26a38c419b1381b8593b0bb6b5d1f056938b086d362/lockfile-0.12.2-py2.py3-none-any.whl
#11 11.73   Saved ./lockfile-0.12.2-py2.py3-none-any.whl
#11 11.73 Collecting Logbook==1.4.3 (from Flask-Core==0.4.0)
#11 11.85   Using cached https://files.pythonhosted.org/packages/f6/83/20fc0270614919cb799f76e32cf143a54c58ce2fa45c19fd38ac2e4f9977/Logbook-1.4.3.tar.gz
#11 12.07 Collecting loggly-python-handler==1.0.0 (from Flask-Core==0.4.0)
#11 12.17   Using cached https://files.pythonhosted.org/packages/00/c5/009460b8c2b82e044121af6e05297270c529b9ce267f499b1ca20330664d/loggly-python-handler-1.0.0.tar.gz
#11 12.35 Collecting mailtrap==0.1.2 (from Flask-Core==0.4.0)
#11 12.37   Using cached https://files.pythonhosted.org/packages/17/c5/ba0fd8287a0eea29033b1c2d835f16d288f82b6601a2c44ca598624d9281/mailtrap-0.1.2-py3-none-any.whl
#11 12.39   Saved ./mailtrap-0.1.2-py3-none-any.whl
#11 12.40 Collecting mandrill==1.0.57 (from Flask-Core==0.4.0)
#11 12.50   Using cached https://files.pythonhosted.org/packages/48/fd/ec2f95504966874bb67b5639592103b77e536cf6ada160cdde5500ae8af6/mandrill-1.0.57.tar.gz
#11 12.69 Collecting MarkupSafe==1.1.1 (from Flask-Core==0.4.0)
#11 12.73   Using cached https://files.pythonhosted.org/packages/b9/2e/64db92e53b86efccfaea71321f597fa2e1b2bd3853d8ce658568f7a13094/MarkupSafe-1.1.1.tar.gz
#11 12.91 Collecting marshmallow==2.19.2 (from Flask-Core==0.4.0)
#11 12.98   Using cached https://files.pythonhosted.org/packages/bc/b4/f54f8533aad0645431bfda2cc3d20913a305779c80f8cb71229e8b615c5b/marshmallow-2.19.2-py2.py3-none-any.whl
#11 12.99   Saved ./marshmallow-2.19.2-py2.py3-none-any.whl
#11 13.00 Collecting marshmallow-sqlalchemy==0.16.2 (from Flask-Core==0.4.0)
#11 13.10   Using cached https://files.pythonhosted.org/packages/56/f6/49e6e0861f28a3250817bb90eed155d538cd860140a90c9c80c40a4a28e1/marshmallow_sqlalchemy-0.16.2-py2.py3-none-any.whl
#11 13.11   Saved ./marshmallow_sqlalchemy-0.16.2-py2.py3-none-any.whl
#11 13.12 Collecting mock==2.0.0 (from Flask-Core==0.4.0)
#11 13.15   Using cached https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl
#11 13.15   Saved ./mock-2.0.0-py2.py3-none-any.whl
#11 13.17 Collecting mongoengine==0.17.0 (from Flask-Core==0.4.0)
#11 13.27   Using cached https://files.pythonhosted.org/packages/5a/1d/84c27ab021df3e0b1d4f3f4258e3ba2c9dac11e381087dfe44768cc65955/mongoengine-0.17.0.tar.gz
#11 13.48 Collecting mongomock==3.16.0 (from Flask-Core==0.4.0)
#11 13.51   Using cached https://files.pythonhosted.org/packages/d6/5c/4b82b7516a414ada96a17b5f245b10cf18e5f768c1fba2142e627cf7f1a2/mongomock-3.16.0-py2.py3-none-any.whl
#11 13.51   Saved ./mongomock-3.16.0-py2.py3-none-any.whl
#11 13.51 Collecting mysql-connector-python==8.0.16 (from Flask-Core==0.4.0)
#11 13.56   Using cached https://files.pythonhosted.org/packages/c5/40/c7ec860ad658870296c1a40afebce8e148997f6759d44307c7b6faa2b130/mysql_connector_python-8.0.16-py2.py3-none-any.whl
#11 13.57   Saved ./mysql_connector_python-8.0.16-py2.py3-none-any.whl
#11 13.58 Collecting nose==1.3.7 (from Flask-Core==0.4.0)
#11 13.60   Using cached https://files.pythonhosted.org/packages/15/d8/dd071918c040f50fa1cf80da16423af51ff8ce4a0f2399b7bf8de45ac3d9/nose-1.3.7-py3-none-any.whl
#11 13.61   Saved ./nose-1.3.7-py3-none-any.whl
#11 13.61 Collecting passlib==1.7.1 (from Flask-Core==0.4.0)
#11 13.63   Using cached https://files.pythonhosted.org/packages/ee/a7/d6d238d927df355d4e4e000670342ca4705a72f0bf694027cf67d9bcf5af/passlib-1.7.1-py2.py3-none-any.whl
#11 13.66   Saved ./passlib-1.7.1-py2.py3-none-any.whl
#11 13.66 Collecting pbr==5.1.3 (from Flask-Core==0.4.0)
#11 13.71   Using cached https://files.pythonhosted.org/packages/14/09/12fe9a14237a6b7e0ba3a8d6fcf254bf4b10ec56a0185f73d651145e9222/pbr-5.1.3-py2.py3-none-any.whl
#11 13.71   Saved ./pbr-5.1.3-py2.py3-none-any.whl
#11 13.72 Collecting postmarker==0.13.0 (from Flask-Core==0.4.0)
#11 13.82   Using cached https://files.pythonhosted.org/packages/88/f5/c64ede9ec785afdcb62b412b47c4ef444479ff4337b270c2626a8c59eba6/postmarker-0.13.0-py2.py3-none-any.whl
#11 13.82   Saved ./postmarker-0.13.0-py2.py3-none-any.whl
#11 13.82 Collecting protobuf==3.7.1 (from Flask-Core==0.4.0)
#11 13.90   Using cached https://files.pythonhosted.org/packages/f5/e4/7c86c9c9b4bad7632e107064fb07d0812b9858a49830d0b2477081bf1267/protobuf-3.7.1-py2.py3-none-any.whl
#11 13.92   Saved ./protobuf-3.7.1-py2.py3-none-any.whl
#11 13.92 Collecting psutil==5.6.2 (from Flask-Core==0.4.0)
#11 14.12   Using cached https://files.pythonhosted.org/packages/c6/c1/beed5e4eaa1345901b595048fab1c85aee647ea0fc02d9e8bf9aceb81078/psutil-5.6.2.tar.gz
#11 14.38 Collecting psycopg2==2.8.2 (from Flask-Core==0.4.0)
#11 14.48   Using cached https://files.pythonhosted.org/packages/23/7e/93c325482c328619870b6cd09370f6dbe1148283daca65115cd63642e60f/psycopg2-2.8.2.tar.gz
#11 14.72 Collecting pycparser==2.19 (from Flask-Core==0.4.0)
#11 14.75   Saved ./pycparser-2.19-py2.py3-none-any.whl
#11 14.75 Collecting pymongo==3.8.0 (from Flask-Core==0.4.0)
#11 15.15   Using cached https://files.pythonhosted.org/packages/19/77/da358f5729ff046ceaf6c6a86755f9d8285719f93df6da15bb2440367d7e/pymongo-3.8.0.tar.gz
#11 15.43 Collecting PyMySQL==0.9.3 (from Flask-Core==0.4.0)
#11 15.46   Using cached https://files.pythonhosted.org/packages/ed/39/15045ae46f2a123019aa968dfcba0396c161c20f855f11dea6796bcaae95/PyMySQL-0.9.3-py2.py3-none-any.whl
#11 15.46   Saved ./PyMySQL-0.9.3-py2.py3-none-any.whl
#11 15.47 Collecting pyScss==1.3.5 (from Flask-Core==0.4.0)
#11 15.56   Using cached https://files.pythonhosted.org/packages/01/7b/c6bfb2515ed08cbfb76b0e72254f24caf76f25676d72024837a85a1e68f5/pyScss-1.3.5.tar.gz
#11 15.76 Collecting python-daemon==2.2.3 (from Flask-Core==0.4.0)
#11 15.79   Using cached https://files.pythonhosted.org/packages/e6/44/6734382d1beb8704137ec94589e7d173a415f3396e34d020fb6b6271c2ff/python_daemon-2.2.3-py2.py3-none-any.whl
#11 15.79   Saved ./python_daemon-2.2.3-py2.py3-none-any.whl
#11 15.79 Collecting python-dateutil==2.8.0 (from Flask-Core==0.4.0)
#11 15.82   Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
#11 15.83   Saved ./python_dateutil-2.8.0-py2.py3-none-any.whl
#11 15.83 Collecting python-dotenv==0.10.1 (from Flask-Core==0.4.0)
#11 15.86   Using cached https://files.pythonhosted.org/packages/8c/14/501508b016e7b1ad0eb91bba581e66ad9bfc7c66fcacbb580eaf9bc38458/python_dotenv-0.10.1-py2.py3-none-any.whl
#11 15.86   Saved ./python_dotenv-0.10.1-py2.py3-none-any.whl
#11 15.87 Collecting python-statsd==2.1.0 (from Flask-Core==0.4.0)
#11 15.89   Using cached https://files.pythonhosted.org/packages/6f/af/224e67da2deda72317aa9afe3445cf849ebd86f4001e20b5d164ef9d05f8/python_statsd-2.1.0-py2.py3-none-any.whl
#11 15.89   Saved ./python_statsd-2.1.0-py2.py3-none-any.whl
#11 15.90 Collecting pytz==2019.1 (from Flask-Core==0.4.0)
#11 16.03   Using cached https://files.pythonhosted.org/packages/3d/73/fe30c2daaaa0713420d0382b16fbb761409f532c56bdcc514bf7b6262bb6/pytz-2019.1-py2.py3-none-any.whl
#11 16.10   Saved ./pytz-2019.1-py2.py3-none-any.whl
#11 16.10 Collecting PyYAML==5.1 (from Flask-Core==0.4.0)
#11 16.14   Using cached https://files.pythonhosted.org/packages/9f/2c/9417b5c774792634834e730932745bc09a7d36754ca00acf1ccd1ac2594d/PyYAML-5.1.tar.gz
#11 16.42 Collecting requests==2.21.0 (from Flask-Core==0.4.0)
#11 16.49   Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl
#11 16.49   Saved ./requests-2.21.0-py2.py3-none-any.whl
#11 16.50 Collecting requests-futures==0.9.9 (from Flask-Core==0.4.0)
#11 16.55   Using cached https://files.pythonhosted.org/packages/e5/6b/c29ba0ce8d7d981c8099550148755846476c551f9e413801c0981f63ea91/requests-futures-0.9.9.tar.gz
#11 16.73 Collecting sentinels==1.0.0 (from Flask-Core==0.4.0)
#11 16.75   Using cached https://files.pythonhosted.org/packages/ac/b7/1af07a98390aba07da31807f3723e7bbd003d6441b4b3d67b20d97702b23/sentinels-1.0.0.tar.gz
#11 16.94 Collecting six==1.12.0 (from Flask-Core==0.4.0)
#11 16.97   Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
#11 16.97   Saved ./six-1.12.0-py2.py3-none-any.whl
#11 16.97 Collecting soupsieve==1.8 (from Flask-Core==0.4.0)
#11 17.00   Using cached https://files.pythonhosted.org/packages/77/78/bca00cc9fa70bba1226ee70a42bf375c4e048fe69066a0d9b5e69bc2a79a/soupsieve-1.8-py2.py3-none-any.whl
#11 17.01   Saved ./soupsieve-1.8-py2.py3-none-any.whl
#11 17.02 Collecting speaklater==1.3 (from Flask-Core==0.4.0)
#11 17.03   Using cached https://files.pythonhosted.org/packages/11/92/5ae1effe0ccb8561c034a0111d53c8788660ddb7ed4992f0da1bb5c525e5/speaklater-1.3.tar.gz
#11 17.22 Collecting SQLAlchemy==1.3.3 (from Flask-Core==0.4.0)
#11 17.30   Using cached https://files.pythonhosted.org/packages/2b/b2/e6f5c5efc68942edefaa924e8fbea0b32375baa434a511cbf6bb17769cf6/SQLAlchemy-1.3.3.tar.gz
#11 18.07 Collecting urllib3==1.25.1 (from Flask-Core==0.4.0)
#11 18.12   Using cached https://files.pythonhosted.org/packages/c0/1f/516c14fd47ced1a2e2882edd776241c5b707ffc9051cd372843579829994/urllib3-1.25.1-py2.py3-none-any.whl
#11 18.13   Saved ./urllib3-1.25.1-py2.py3-none-any.whl
#11 18.14 Collecting vine==1.3.0 (from Flask-Core==0.4.0)
#11 18.23   Using cached https://files.pythonhosted.org/packages/7f/60/82c03047396126c8331ceb64da1dc52d4f1317209f32e8fe286d0c07365a/vine-1.3.0-py2.py3-none-any.whl
#11 18.24   Saved ./vine-1.3.0-py2.py3-none-any.whl
#11 18.24 Collecting webassets==0.12.1 (from Flask-Core==0.4.0)
#11 18.33   Using cached https://files.pythonhosted.org/packages/9a/35/d86c6d3bd31e2e9c8351c8766ef3688c75d56bc5233ed49061e522efc3cc/webassets-0.12.1.tar.gz
#11 18.57 Collecting webencodings==0.5.1 (from Flask-Core==0.4.0)
#11 18.59   Using cached https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl
#11 18.59   Saved ./webencodings-0.5.1-py2.py3-none-any.whl
#11 18.59 Collecting WebOb==1.8.5 (from Flask-Core==0.4.0)
#11 18.63   Using cached https://files.pythonhosted.org/packages/06/e1/4acd2b4327fceb4c6446bdbca515f807ab83188526fd654940c00bcf8cc3/WebOb-1.8.5-py2.py3-none-any.whl
#11 18.64   Saved ./WebOb-1.8.5-py2.py3-none-any.whl
#11 18.64 Collecting Werkzeug==0.15.2 (from Flask-Core==0.4.0)
#11 18.68   Using cached https://files.pythonhosted.org/packages/18/79/84f02539cc181cdbf5ff5a41b9f52cae870b6f632767e43ba6ac70132e92/Werkzeug-0.15.2-py2.py3-none-any.whl
#11 18.70   Saved ./Werkzeug-0.15.2-py2.py3-none-any.whl
#11 18.71 Collecting WTForms==2.2.1 (from Flask-Core==0.4.0)
#11 18.73   Using cached https://files.pythonhosted.org/packages/9f/c8/dac5dce9908df1d9d48ec0e26e2a250839fa36ea2c602cc4f85ccfeb5c65/WTForms-2.2.1-py2.py3-none-any.whl
#11 18.74   Saved ./WTForms-2.2.1-py2.py3-none-any.whl
#11 18.75 Collecting setuptools (from protobuf==3.7.1->Flask-Core==0.4.0)
#11 19.01   Using cached https://files.pythonhosted.org/packages/ec/51/f45cea425fd5cb0b0380f5b0f048ebc1da5b417e48d304838c02d6288a1e/setuptools-41.0.1-py2.py3-none-any.whl
#11 19.04   Saved ./setuptools-41.0.1-py2.py3-none-any.whl
#11 19.04 Skipping amqp, due to already being wheel.
#11 19.04 Skipping asn1crypto, due to already being wheel.
#11 19.04 Skipping beautifulsoup4, due to already being wheel.
#11 19.04 Skipping bugsnag, due to already being wheel.
#11 19.04 Skipping certifi, due to already being wheel.
#11 19.04 Skipping cffi, due to already being wheel.
#11 19.04 Skipping chardet, due to already being wheel.
#11 19.04 Skipping Click, due to already being wheel.
#11 19.04 Skipping cssutils, due to already being wheel.
#11 19.04 Skipping docutils, due to already being wheel.
#11 19.04 Skipping Flask, due to already being wheel.
#11 19.04 Skipping Flask-SQLAlchemy, due to already being wheel.
#11 19.04 Skipping Flask-WTF, due to already being wheel.
#11 19.04 Skipping gevent-socketio-hartwork, due to already being wheel.
#11 19.04 Skipping gevent-websocket, due to already being wheel.
#11 19.04 Skipping html5lib, due to already being wheel.
#11 19.04 Skipping idna, due to already being wheel.
#11 19.04 Skipping itsdangerous, due to already being wheel.
#11 19.04 Skipping Jinja2, due to already being wheel.
#11 19.04 Skipping kombu, due to already being wheel.
#11 19.04 Skipping lockfile, due to already being wheel.
#11 19.04 Skipping mailtrap, due to already being wheel.
#11 19.04 Skipping marshmallow, due to already being wheel.
#11 19.04 Skipping marshmallow-sqlalchemy, due to already being wheel.
#11 19.05 Skipping mock, due to already being wheel.
#11 19.05 Skipping mongomock, due to already being wheel.
#11 19.05 Skipping mysql-connector-python, due to already being wheel.
#11 19.05 Skipping nose, due to already being wheel.
#11 19.05 Skipping passlib, due to already being wheel.
#11 19.05 Skipping pbr, due to already being wheel.
#11 19.05 Skipping postmarker, due to already being wheel.
#11 19.05 Skipping protobuf, due to already being wheel.
#11 19.05 Skipping pycparser, due to already being wheel.
#11 19.05 Skipping PyMySQL, due to already being wheel.
#11 19.05 Skipping python-daemon, due to already being wheel.
#11 19.05 Skipping python-dateutil, due to already being wheel.
#11 19.05 Skipping python-dotenv, due to already being wheel.
#11 19.05 Skipping python-statsd, due to already being wheel.
#11 19.05 Skipping pytz, due to already being wheel.
#11 19.05 Skipping requests, due to already being wheel.
#11 19.05 Skipping six, due to already being wheel.
#11 19.05 Skipping soupsieve, due to already being wheel.
#11 19.05 Skipping urllib3, due to already being wheel.
#11 19.05 Skipping vine, due to already being wheel.
#11 19.05 Skipping webencodings, due to already being wheel.
#11 19.05 Skipping WebOb, due to already being wheel.
#11 19.05 Skipping Werkzeug, due to already being wheel.
#11 19.05 Skipping WTForms, due to already being wheel.
#11 19.05 Skipping setuptools, due to already being wheel.
#11 19.05 Building wheels for collected packages: Flask-Core, amqplib, anyjson, billiard, blinker, configobj, cryptography, cssmin, cssprefixer, docopt, Flask-Assets, flask-mongoengine, Flask-Script, gevent, greenlet, Logbook, loggly-python-handler, mandrill, MarkupSafe, mongoengine, psutil, psycopg2, pymongo, pyScss, PyYAML, requests-futures, sentinels, speaklater, SQLAlchemy, webassets
#11 19.05   Building wheel for Flask-Core (setup.py): started
#11 19.40   Building wheel for Flask-Core (setup.py): finished with status 'done'
#11 19.40   Stored in directory: /src
#11 19.41   Building wheel for amqplib (setup.py): started
#11 20.34   Building wheel for amqplib (setup.py): finished with status 'done'
#11 20.34   Stored in directory: /src
#11 20.34   Building wheel for anyjson (setup.py): started
#11 20.86   Building wheel for anyjson (setup.py): finished with status 'done'
#11 20.86   Stored in directory: /src
#11 20.86   Building wheel for billiard (setup.py): started
#11 21.11   Building wheel for billiard (setup.py): finished with status 'done'
#11 21.11   Stored in directory: /src
#11 21.11   Building wheel for blinker (setup.py): started
#11 21.33   Building wheel for blinker (setup.py): finished with status 'done'
#11 21.33   Stored in directory: /src
#11 21.33   Building wheel for configobj (setup.py): started
#11 21.55   Building wheel for configobj (setup.py): finished with status 'done'
#11 21.55   Stored in directory: /src
#11 21.55   Building wheel for cryptography (PEP 517): started
#11 41.32   Building wheel for cryptography (PEP 517): finished with status 'done'
#11 41.32   Stored in directory: /src
#11 41.32   Building wheel for cssmin (setup.py): started
#11 41.53   Building wheel for cssmin (setup.py): finished with status 'done'
#11 41.53   Stored in directory: /src
#11 41.53   Building wheel for cssprefixer (setup.py): started
#11 41.74   Building wheel for cssprefixer (setup.py): finished with status 'done'
#11 41.74   Stored in directory: /src
#11 41.74   Building wheel for docopt (setup.py): started
#11 41.95   Building wheel for docopt (setup.py): finished with status 'done'
#11 41.95   Stored in directory: /src
#11 41.95   Building wheel for Flask-Assets (setup.py): started
#11 42.17   Building wheel for Flask-Assets (setup.py): finished with status 'done'
#11 42.17   Stored in directory: /src
#11 42.17   Building wheel for flask-mongoengine (setup.py): started
#11 42.45   Building wheel for flask-mongoengine (setup.py): finished with status 'done'
#11 42.45   Stored in directory: /src
#11 42.45   Building wheel for Flask-Script (setup.py): started
#11 42.66   Building wheel for Flask-Script (setup.py): finished with status 'done'
#11 42.67   Stored in directory: /src
#11 42.67   Building wheel for gevent (setup.py): started
#11 103.4   Building wheel for gevent (setup.py): still running...
#11 124.4   Building wheel for gevent (setup.py): finished with status 'done'
#11 124.4   Stored in directory: /src
#11 124.4   Building wheel for greenlet (setup.py): started
#11 125.1   Building wheel for greenlet (setup.py): finished with status 'done'
#11 125.1   Stored in directory: /src
#11 125.1   Building wheel for Logbook (setup.py): started
#11 125.4   Building wheel for Logbook (setup.py): finished with status 'done'
#11 125.4   Stored in directory: /src
#11 125.4   Building wheel for loggly-python-handler (setup.py): started
#11 125.6   Building wheel for loggly-python-handler (setup.py): finished with status 'done'
#11 125.6   Stored in directory: /src
#11 125.6   Building wheel for mandrill (setup.py): started
#11 125.8   Building wheel for mandrill (setup.py): finished with status 'done'
#11 125.8   Stored in directory: /src
#11 125.8   Building wheel for MarkupSafe (setup.py): started
#11 126.2   Building wheel for MarkupSafe (setup.py): finished with status 'done'
#11 126.2   Stored in directory: /src
#11 126.2   Building wheel for mongoengine (setup.py): started
#11 129.2   Building wheel for mongoengine (setup.py): finished with status 'done'
#11 129.2   Stored in directory: /src
#11 129.2   Building wheel for psutil (setup.py): started
#11 130.2   Building wheel for psutil (setup.py): finished with status 'done'
#11 130.2   Stored in directory: /src
#11 130.2   Building wheel for psycopg2 (setup.py): started
#11 136.4   Building wheel for psycopg2 (setup.py): finished with status 'done'
#11 136.4   Stored in directory: /src
#11 136.4   Building wheel for pymongo (setup.py): started
#11 138.8   Building wheel for pymongo (setup.py): finished with status 'done'
#11 138.8   Stored in directory: /src
#11 138.8   Building wheel for pyScss (setup.py): started
#11 139.1   Building wheel for pyScss (setup.py): finished with status 'done'
#11 139.1   Stored in directory: /src
#11 139.1   Building wheel for PyYAML (setup.py): started
#11 139.4   Building wheel for PyYAML (setup.py): finished with status 'done'
#11 139.4   Stored in directory: /src
#11 139.4   Building wheel for requests-futures (setup.py): started
#11 139.6   Building wheel for requests-futures (setup.py): finished with status 'done'
#11 139.6   Stored in directory: /src
#11 139.6   Building wheel for sentinels (setup.py): started
#11 139.8   Building wheel for sentinels (setup.py): finished with status 'done'
#11 139.8   Stored in directory: /src
#11 139.8   Building wheel for speaklater (setup.py): started
#11 140.0   Building wheel for speaklater (setup.py): finished with status 'done'
#11 140.0   Stored in directory: /src
#11 140.0   Building wheel for SQLAlchemy (setup.py): started
#11 141.1   Building wheel for SQLAlchemy (setup.py): finished with status 'done'
#11 141.1   Stored in directory: /src
#11 141.1   Building wheel for webassets (setup.py): started
#11 141.3   Building wheel for webassets (setup.py): finished with status 'done'
#11 141.3   Stored in directory: /src
#11 141.3 Successfully built Flask-Core amqplib anyjson billiard blinker configobj cryptography cssmin cssprefixer docopt Flask-Assets flask-mongoengine Flask-Script gevent greenlet Logbook loggly-python-handler mandrill MarkupSafe mongoengine psutil psycopg2 pymongo pyScss PyYAML requests-futures sentinels speaklater SQLAlchemy webassets
#11 141.6 WARNING: You are using pip version 19.1.1, however version 19.2.1 is available.
#11 141.6 You should consider upgrading via the 'pip install --upgrade pip' command.
#11 DONE 141.7s

#12 exporting to image
#12 exporting layers
#12 exporting layers 0.2s done
#12 writing image sha256:746048384e8d72ec622e1331a469752d132f43c5657ab05aff27740cba65efec done
FernandoMiguel commented 5 years ago

here is docker-compose second run

docker-compose -f .jenkins/docker-compose.images-base.1.yml build flask-core
Building flask-core
Step 1/5 : FROM python:3.6-alpine AS ms-python-wheel
 ---> f00bde8f9c15
Step 2/5 : COPY . /src/
 ---> Using cache
 ---> a78f716cc093
Step 3/5 : RUN apk add --no-cache --virtual build-dependencies     gcc     build-base     libffi-dev     libressl-dev     postgresql-dev
 ---> Using cache
 ---> e53e0370ed67
Step 4/5 : RUN cd /src && pip wheel -e .
 ---> Using cache
 ---> 68ea54a7cf49
Step 5/5 : WORKDIR /src
 ---> Using cache
 ---> 925337004af3

Successfully built 925337004af3
FernandoMiguel commented 5 years ago

in this screenshot you can see 4 runs. from the bottom up: buildx 1st run buildx 2nd run (should have cached but actually took even longer) docker-compose 1st build with no cache docker-compose 2nd run (everything was cached from the 1st build, so was very quick)

Screenshot 2019-07-28 15 26 50
FernandoMiguel commented 5 years ago 
root@ip-10-50-0-8:~# journalctl -fu docker.service
-- Logs begin at Sun 2019-07-28 11:42:29 UTC. --
Jul 28 13:47:28 ip-10-50-0-8 dockerd[9697]: time="2019-07-28T13:47:28.607056390Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc01d801ec0, READY" module=grpc
Jul 28 13:49:51 ip-10-50-0-8 dockerd[9697]: time="2019-07-28T13:49:51.321901451Z" level=warning msg="grpc: addrConn.createTransport failed to connect to { 0  <nil>}. Err :connection error: desc = \"transport: Error while dialing only one connection allowed\". Reconnecting..." module=grpc
Jul 28 14:09:42 ip-10-50-0-8 dockerd[9697]: time="2019-07-28T14:09:42.050784099Z" level=info msg="parsed scheme: \"\"" module=grpc
Jul 28 14:09:42 ip-10-50-0-8 dockerd[9697]: time="2019-07-28T14:09:42.051186102Z" level=info msg="scheme \"\" not registered, fallback to default scheme" module=grpc
Jul 28 14:09:42 ip-10-50-0-8 dockerd[9697]: time="2019-07-28T14:09:42.051217375Z" level=info msg="ccResolverWrapper: sending update to cc: {[{ 0  <nil>}] }" module=grpc
Jul 28 14:09:42 ip-10-50-0-8 dockerd[9697]: time="2019-07-28T14:09:42.051227436Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jul 28 14:09:42 ip-10-50-0-8 dockerd[9697]: time="2019-07-28T14:09:42.051335775Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc0049ae630, CONNECTING" module=grpc
Jul 28 14:09:42 ip-10-50-0-8 dockerd[9697]: time="2019-07-28T14:09:42.051481428Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc0049ae630, READY" module=grpc
Jul 28 14:12:16 ip-10-50-0-8 dockerd[9697]: time="2019-07-28T14:12:16.987863612Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc0049ae630, TRANSIENT_FAILURE" module=grpc
Jul 28 14:12:16 ip-10-50-0-8 dockerd[9697]: time="2019-07-28T14:12:16.987886888Z" level=warning msg="grpc: addrConn.createTransport failed to connect to { 0  <nil>}. Err :connection error: desc = \"transport: Error while dialing only one connection allowed\". Reconnecting..." module=grpc

Jul 28 14:18:17 ip-10-50-0-8 dockerd[9697]: time="2019-07-28T14:18:17.962045963Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jul 28 14:18:18 ip-10-50-0-8 dockerd[9697]: time="2019-07-28T14:18:18.040995668Z" level=warning msg="5fa0d898c7160c8ab15e044c09dac750a58189f2a826c04c84454de86f8f7779 cleanup: failed to unmount IPC: umount /mnt/nvme/docker/containers/5fa0d898c7160c8ab15e044c09dac750a58189f2a826c04c84454de86f8f7779/mounts/shm, flags: 0x2: no such file or directory"
Jul 28 14:20:53 ip-10-50-0-8 dockerd[9697]: time="2019-07-28T14:20:53.738637381Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jul 28 14:20:53 ip-10-50-0-8 dockerd[9697]: time="2019-07-28T14:20:53.840989639Z" level=warning msg="0f45a625e93f0d986a26954e0daf3ebee9ff888f25d720c0ae37c06dca822ec1 cleanup: failed to unmount IPC: umount /mnt/nvme/docker/containers/0f45a625e93f0d986a26954e0daf3ebee9ff888f25d720c0ae37c06dca822ec1/mounts/shm, flags: 0x2: no such file or directory"
Jul 28 14:20:54 ip-10-50-0-8 dockerd[9697]: time="2019-07-28T14:20:54.441387293Z" level=info msg="Layer sha256:0cd43e560d4f95f90e9aeec0a4a0c67aee0824a192a2a6d7d39b1b4b1ddda03d cleaned up"
tonistiigi commented 5 years ago

Am I right to assume that these builds happen from different checkouts? A common thing that is not guaranteed to be deterministic is the .git folder. You can filter the COPY command to only include the files that you need or use add .git to .dockerignore. If you are not sure what gets changed you can print the shasum of all fo the files before the job.

FernandoMiguel commented 5 years ago

@tonistiigi the build setup checkout the same commit id And in this case I even disabled all the checkouts so it would use the same files on disk All repos have a .dockerignore with .git and Dockerfile/buildx listed inside

This is just one example, and I have have several more services doing the same.

tonistiigi commented 5 years ago

@FernandoMiguel Can you post an example that is reproducible?

FernandoMiguel commented 5 years ago

@tonistiigi since I'm running this against our organisation code base, I can't share that code. But I'll try to mimic the build behaviour with an hello world example. I'll algo get the file hashes just to double check the build context is the same. Is there a way to see from the Docker agent if the context hasn't change?

tonistiigi commented 5 years ago

As this seems to be the last layer that loses the cache it might also be that is gets cleaned up by gc. You can try to turn it off of set a bigger limit. If you run with debug logs you can see the gc invocations from the logs.

FernandoMiguel commented 5 years ago

@tonistiigi I've seen it happen with intermediary layers too.

How can I increase the cache? We are talking of about 17GB of data (looking at system prune output). These builders come with 150GB of ephemeral SSD, so it ain't a problem

tonistiigi commented 5 years ago

@FernandoMiguel --oci-worker-gc=false to turn off --oci-worker-gc-keepstorage= to control it (more detailed control with the config file). BuildKit is conservative by default and doesn't want to eat away your whole disk. Also, I can't see that driver/exporter you are using. If you export to the docker image store than this data is not released automatically and it can't be the issue then. If you use docker driver then gc is not enabled by default.

FernandoMiguel commented 5 years ago

cheers @tonistiigi

FernandoMiguel commented 5 years ago

I can't see that driver/exporter you are using. If you export to the docker image store than this data is not released automatically and it can't be the issue then. If you use docker driver then gc is not enabled by default.

it exports to the local docker registry on the host. docker-compose push is used later to push all images of the build

FernandoMiguel commented 5 years ago

i spent a while on this, but can't find a way to enable debug or gc=false, when using buildx bake some guidance would be appreciated

tonistiigi commented 5 years ago

https://github.com/docker/buildx/pull/102

FernandoMiguel commented 5 years ago

@tonistiigi thanks. I tried that, but buildx inspect doesn't show that flag And docker logs of the buildkit id is very quiet too

tonistiigi commented 5 years ago

I don't know what you mean by inspect, the flag is for create. This PR is not in a release build, you would need to build it yourself.

FernandoMiguel commented 5 years ago

@tonistiigi I see. Thanks. I'll give it another go tomorrow

FernandoMiguel commented 5 years ago 
/usr/libexec/docker/cli-plugins/docker-buildx build --platform=local -o . git://github.com/docker/buildx
/usr/libexec/docker/cli-plugins/docker-buildx create --use --buildkitd-flags '--debug --oci-worker-gc=false'
 /usr/libexec/docker/cli-plugins/docker-buildx inspect quirky_merkle
Name:   quirky_merkle
Driver: docker-container

Nodes:
Name:      quirky_merkle0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Flags:     --debug --oci-worker-gc=false
Platforms: linux/amd64

Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        74b1e89
 Built:             Thu Jul 25 21:21:22 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       74b1e89
  Built:            Thu Jul 25 21:19:53 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Client:
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc.)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 7
 Server Version: 19.03.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.0.0-1010-aws
 Operating System: Ubuntu 19.04
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 30.96GiB
 Name: ip-10-50-0-9
 ID: FWMR:6PLO:IWGI:LHTL:KQXD:2IO2:XCUP:PAYZ:USP7:ZFAN:P575:CIEH
 Docker Root Dir: /mnt/nvme/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

/usr/libexec/docker/cli-plugins/docker-buildx bake --progress plain -f .jenkins/docker-compose.images-base.1.yml -f .jenkins/buildx.images-base.1.yml flask
time="2019-07-31T18:55:55Z" level=debug msg="session started"
time="2019-07-31T18:55:55Z" level=debug msg="reusing ref for local: 0tow3mjblxanl0jerwyu3gaqq"
time="2019-07-31T18:55:55Z" level=debug msg="reusing ref for local: 1qk8dru7hiwl0xmzjr4fkn43z"
time="2019-07-31T18:55:55Z" level=debug msg="diffcopy took: 2.642166ms"
time="2019-07-31T18:55:55Z" level=debug msg="diffcopy took: 3.103725ms"
time="2019-07-31T18:55:55Z" level=debug msg="saved 1qk8dru7hiwl0xmzjr4fkn43z as local.sharedKey:context:context-.dockerignore:"
time="2019-07-31T18:55:55Z" level=debug msg="saved 0tow3mjblxanl0jerwyu3gaqq as local.sharedKey:dockerfile:dockerfile:"
time="2019-07-31T18:55:55Z" level=debug msg=resolving
time="2019-07-31T18:55:55Z" level=debug msg="do request" request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] User-Agent:[containerd/1.2.0+unknown]]" request.method=HEAD url="https://registry-1.docker.io/v2/docker/dockerfile/manifests/experimental"
time="2019-07-31T18:55:55Z" level=debug msg="fetch response received" response.headers="map[Content-Length:[160] Content-Type:[application/json; charset=utf-8] Date:[Wed, 31 Jul 2019 18:55:55 GMT] Docker-Distribution-Api-Version:[registry/2.0] Strict-Transport-Security:[max-age=31536000] Www-Authenticate:[Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:docker/dockerfile:pull\"]]" status="401 Unauthorized" url="https://registry-1.docker.io/v2/docker/dockerfile/manifests/experimental"
time="2019-07-31T18:55:55Z" level=debug msg=Unauthorized header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:docker/dockerfile:pull\""
time="2019-07-31T18:55:55Z" level=debug msg="do request" request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] User-Agent:[containerd/1.2.0+unknown]]" request.method=HEAD url="https://registry-1.docker.io/v2/docker/dockerfile/manifests/experimental"
time="2019-07-31T18:55:55Z" level=debug msg="fetch response received" response.headers="map[Content-Length:[2029] Content-Type:[application/vnd.docker.distribution.manifest.list.v2+json] Date:[Wed, 31 Jul 2019 18:55:55 GMT] Docker-Content-Digest:[sha256:9022e911101f01b2854c7a4b2c77f524b998891941da55208e71c0335e6e82c3] Docker-Distribution-Api-Version:[registry/2.0] Etag:[\"sha256:9022e911101f01b2854c7a4b2c77f524b998891941da55208e71c0335e6e82c3\"] Strict-Transport-Security:[max-age=31536000]]" status="200 OK" url="https://registry-1.docker.io/v2/docker/dockerfile/manifests/experimental"
time="2019-07-31T18:55:55Z" level=debug msg=resolved desc.digest="sha256:9022e911101f01b2854c7a4b2c77f524b998891941da55208e71c0335e6e82c3"
time="2019-07-31T18:55:55Z" level=debug msg=fetch digest="sha256:9022e911101f01b2854c7a4b2c77f524b998891941da55208e71c0335e6e82c3" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=2029
time="2019-07-31T18:55:55Z" level=debug msg=fetch digest="sha256:962337bd2c94ed20329caf3201963fd81d2892544c4502f1b237f5063e0fe27f" mediatype=application/vnd.docker.distribution.manifest.v2+json size=527
time="2019-07-31T18:55:55Z" level=debug msg=fetch digest="sha256:8bf79a4e725097175c47f89c7b1efffe300bb4fd4e9ac45eea9daba66f81e904" mediatype=application/vnd.docker.container.image.v1+json size=897
time="2019-07-31T18:55:55Z" level=debug msg="load cache for docker-image://docker.io/docker/dockerfile:experimental@sha256:9022e911101f01b2854c7a4b2c77f524b998891941da55208e71c0335e6e82c3 with 9ys7i6me1k7hns19i191htcky::sha256:d469e8fdff2f0208fb8c23b518279d9d73e5efc9e10be65e71942fc72c38e31a"
time="2019-07-31T18:55:55Z" level=debug msg="serving grpc connection"
time="2019-07-31T18:55:55Z" level=debug msg="> creating ja4kewo4za8zm8824biees2ld [/bin/dockerfile-frontend]"
time="2019-07-31T18:55:56Z" level=debug msg=resolving
time="2019-07-31T18:55:56Z" level=debug msg="do request" request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] User-Agent:[containerd/1.2.0+unknown]]" request.method=HEAD url="https://registry-1.docker.io/v2/library/python/manifests/3.6-alpine"
time="2019-07-31T18:55:56Z" level=debug msg="fetch response received" response.headers="map[Content-Length:[157] Content-Type:[application/json; charset=utf-8] Date:[Wed, 31 Jul 2019 18:55:56 GMT] Docker-Distribution-Api-Version:[registry/2.0] Strict-Transport-Security:[max-age=31536000] Www-Authenticate:[Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/python:pull\"]]" status="401 Unauthorized" url="https://registry-1.docker.io/v2/library/python/manifests/3.6-alpine"
time="2019-07-31T18:55:56Z" level=debug msg=Unauthorized header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/python:pull\""
time="2019-07-31T18:55:56Z" level=debug msg="do request" request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] User-Agent:[containerd/1.2.0+unknown]]" request.method=HEAD url="https://registry-1.docker.io/v2/library/python/manifests/3.6-alpine"
time="2019-07-31T18:55:56Z" level=debug msg="fetch response received" response.headers="map[Content-Length:[1645] Content-Type:[application/vnd.docker.distribution.manifest.list.v2+json] Date:[Wed, 31 Jul 2019 18:55:56 GMT] Docker-Content-Digest:[sha256:cea913d3f439c10cd67f274b7cb09bb76c8b167113c8339a5c201ee491716d55] Docker-Distribution-Api-Version:[registry/2.0] Etag:[\"sha256:cea913d3f439c10cd67f274b7cb09bb76c8b167113c8339a5c201ee491716d55\"] Strict-Transport-Security:[max-age=31536000]]" status="200 OK" url="https://registry-1.docker.io/v2/library/python/manifests/3.6-alpine"
time="2019-07-31T18:55:56Z" level=debug msg=resolved desc.digest="sha256:cea913d3f439c10cd67f274b7cb09bb76c8b167113c8339a5c201ee491716d55"
time="2019-07-31T18:55:56Z" level=debug msg=fetch digest="sha256:cea913d3f439c10cd67f274b7cb09bb76c8b167113c8339a5c201ee491716d55" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=1645
time="2019-07-31T18:55:56Z" level=debug msg=fetch digest="sha256:869690514dc569fbda884f6ee74fef8f602ac17d0dd369f661936a87747593f5" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1368
time="2019-07-31T18:55:56Z" level=debug msg=fetch digest="sha256:dd987e41d7d970c82d5df98bd8914fcb620368aac8edfcb1ea7902dfe79d2419" mediatype=application/vnd.docker.container.image.v1+json size=7620
time="2019-07-31T18:55:56Z" level=debug msg="reusing ref for local: 4che3w9k9w56ltrti5mp4wl2j"
time="2019-07-31T18:55:56Z" level=debug msg="diffcopy took: 10.868491ms"
time="2019-07-31T18:55:56Z" level=debug msg="saved 4che3w9k9w56ltrti5mp4wl2j as local.sharedKey:context:context:"
time="2019-07-31T18:55:56Z" level=debug msg="load cache for [3/5] RUN --mount=type=cache,id=apk,sharing=locked,target=/var/cache/apk ln -vs /var/cache/apk /etc/apk/cache && \tapk add --update --virtual build-dependencies \tgcc \tbuild-base \tlibffi-dev \tlibressl-dev \tpostgresql-dev with 9ys7i6me1k7hns19i191htcky::1jb667nqcb5stxrdhz7ij42av"
time="2019-07-31T18:55:56Z" level=debug msg="reusing ref for cache dir: ga3mk0fht5ywu4omjqfhxfs1t"
time="2019-07-31T18:55:56Z" level=debug msg="> creating axgr1ai8l77m6usy1xwegp8ah [/bin/sh -c pip wheel -e .]"
time="2019-07-31T18:58:20Z" level=debug msg="session finished: <nil>"
time="2019-07-31T18:58:21Z" level=debug msg="remove snapshot" key=84r0ia7z8euguohd9tufpc3kp snapshotter=overlayfs
time="2019-07-31T18:58:21Z" level=debug msg="schedule snapshotter cleanup" snapshotter=overlayfs
time="2019-07-31T18:58:21Z" level=debug msg="removed snapshot" key=buildkit/1329/84r0ia7z8euguohd9tufpc3kp snapshotter=overlayfs
time="2019-07-31T18:58:21Z" level=debug msg="snapshot garbage collected" d=5.255037ms snapshotter=overlayfs
/usr/libexec/docker/cli-plugins/docker-buildx bake --progress plain -f .jenkins/docker-compose.images-base.1.yml -f .jenkins/buildx.images-base.1.yml
time="2019-07-31T18:55:55Z" level=warning msg="No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load"
#1 [internal] load .dockerignore
#1 transferring context: 71B done
#1 DONE 0.0s

#2 [internal] load build definition from buildx
#2 transferring dockerfile: 449B done
#2 DONE 0.0s

#3 resolve image config for docker.io/docker/dockerfile:experimental
time="2019-07-31T18:55:55Z" level=error msg="Error parsing the serverURL" error="docker-credential-ecr-login can only be used with Amazon Elastic Container Registry." serverURL="https://index.docker.io/v1/"
#3 DONE 0.7s
time="2019-07-31T18:55:56Z" level=error msg="Error parsing the serverURL" error="docker-credential-ecr-login can only be used with Amazon Elastic Container Registry." serverURL="https://index.docker.io/v1/"

#4 docker-image://docker.io/docker/dockerfile:experimental@sha256:9022e9111...
#4 resolve docker.io/docker/dockerfile:experimental@sha256:9022e911101f01b2854c7a4b2c77f524b998891941da55208e71c0335e6e82c3 done
#4 CACHED

#5 [internal] load metadata for docker.io/library/python:3.6-alpine
#5 DONE 0.3s

#6 [1/5] FROM docker.io/library/python:3.6-alpine@sha256:cea913d3f439c10cd6...
#6 resolve docker.io/library/python:3.6-alpine@sha256:cea913d3f439c10cd67f274b7cb09bb76c8b167113c8339a5c201ee491716d55 done
#6 DONE 0.0s

#9 [internal] load build context
#9 transferring context: 91.37kB 0.0s done
#9 DONE 0.0s

#7 [2/5] WORKDIR /src
#7 CACHED

#8 [3/5] RUN --mount=type=cache,id=apk,sharing=locked,target=/var/cache/apk...
#8 CACHED

#10 [4/5] COPY . /src/
#10 DONE 0.0s

#11 [5/5] RUN --mount=type=cache,id=wheel,sharing=locked,target=/root   pip w...
#11 1.054 Obtaining file:///src
FernandoMiguel commented 5 years ago
Screenshot 2019-07-31 19 58 21

this is a docker-compose 1st run, no cache

FernandoMiguel commented 5 years ago
Screenshot 2019-07-31 19 58 39

this is a docker-compose 2nd run, fully cached

FernandoMiguel commented 5 years ago
Screenshot 2019-07-31 19 58 58

this is buildx/BuildKit 1st and 2nd run there is a lot of cache bust on the 2nd run

FernandoMiguel commented 5 years ago 
# syntax=docker/dockerfile:experimental

FROM python:3.6-alpine AS ms-python-wheel

WORKDIR /src

RUN --mount=type=cache,id=apk,sharing=locked,target=/var/cache/apk ln -vs /var/cache/apk /etc/apk/cache && \
    apk add --update --virtual build-dependencies \
    gcc \
    build-base \
    libffi-dev \
    libressl-dev \
    postgresql-dev

COPY . /src/
RUN --mount=type=cache,id=wheel,sharing=locked,target=/root \
    pip wheel -e .
tiborvass commented 5 years ago

@FernandoMiguel from the output, you can see that COPY . step is not cached. So something has changed in your build context. Keep in mind that hidden files/directories are part of the cache. If you can, it is better to specify the files to copy in.

FernandoMiguel commented 5 years ago

@tiborvass yes I'm aware of the risks of using dot, and when possible avoid it. Sadly some applications are too complex to identify all required resources.

These are consequentive runs of the same commit id.

When using docker compose with a similar Dockerfile (minus the buildkit cache stuff) this very same build will be fully cached. Not only this, but the entire set of images and code base.

I would love to dig deeper and understand what is making a cache miss.

If you have any ideas, I would love to see BuildKit beat docker-compose 😊

thaJeztah commented 4 years ago

@FernandoMiguel this still an issue?

FernandoMiguel commented 4 years ago

@thaJeztah i no longer have access to the underlying code base to try to reproduce