search

uber-archive / makisu

Fast and flexible Docker image building tool, works in unprivileged containerized environments like Mesos and Kubernetes.
Apache License 2.0
2.41k stars 154 forks source link

Failed to push cache: timeout waiting for push #247

Open geekflyer opened 5 years ago

geekflyer commented 5 years ago

I'm trying to run makisu on our own Kubernetes cluster via via https://cirrus-ci.com.

While I somewhat got it working, every 5-th build or so get's stuck for a long a time (i.e. about 10 minutes) at a step like this:

1.5643073208686275e+09  info    * Pushing with cache ID eb427ee8
1.5643073208691173e+09  info    * Moving directories [] to /makisu-storage/sandbox/sandbox746334368/stages/MA==

After waiting for 10 minutes the build then usually prints this: 1.5643079235246627e+09 error Failed to push cache: timeout waiting for push

and then continues and successfully completes.

For context:

My suspicion is that this a problem that occurs in conjunction with the GCR credential helper.

Makisu version is: v0.1.11.

Registry config looks like this:

   {
        "gcr.io": {
          "myproject-*/*": {
            "push_chunk": -1,
            "security": {
              "credsStore": "gcr"
            }
          }
        }
      }

Makisu is invoked with this command:


    - /makisu-internal/makisu build
      --modifyfs
      --log-fmt=console
      --http-cache-addr http://$CIRRUS_HTTP_CACHE_HOST/
      --registry-config="${REGISTRY_CONFIG}"
      --commit=explicit
      --push=gcr.io
      --tag=myproject/$APP:$CIRRUS_CHANGE_IN_REPO
      --replica=gcr.io/myproject/$APP:$CIRRUS_CHANGE_IN_REPO
      --file=$CIRRUS_WORKING_DIR/Dockerfile
      $CIRRUS_WORKING_DIR
yiranwang52 commented 5 years ago

Do you know how many layers your images have?

geekflyer commented 5 years ago

24 layers (includes the base layers). With commit=explicit it got down to about 16, but the error still occassionally occurs. The image is based off FROM node:10 but I also got this error a few times with another image which is based FROM python:3.6.6 and has a similar amount of layers.

evelynl94 commented 5 years ago

Do you know how big are the layers? Have you tried increasing the concurrency in registry configuration?

geekflyer commented 5 years ago

The layers are quite big but not huge. Altogether image A is compressed about 400Mb. The other one is even bigger, probably around 1Gig compressed. How would increasing the concurrency help? I'm pretty sure it's not a network / bandwidth issue. We're running the builds inside GKE and push it to GCR. Also, the builds in which this issue doesn't occur usually complete in less than 2 minutes, incl. pushing the layers (as mentioned this issue only occurs on every 5-th build or so).

My guess is honestly that somehow makisu swallows an error (maybe from the credential helper or maybe because of some invalidated token?) and that's why the timeout occurs. I also could imagine that it's somehow related to http cache. Unfortunately the error logs are not very useful with regards to what went wrong. I'll see if I can create a repro of this with a public repo once I got some time.

evelynl94 commented 5 years ago

The registry client is kinda verbose and would fail if credHelper returns an error but default timeout for pushing a layer is also 10 mins (which is too long). Could you provide us the entire build log? Maybe we missed something...

Also (probably unlikely), is it possible that you got rate limited by GCR? https://cloud.google.com/container-registry/quotas

geekflyer commented 5 years ago

Sure,

there you go:

cd resolve
/makisu-internal/makisu build --modifyfs --log-fmt=console --http-cache-addr http://$CIRRUS_HTTP_CACHE_HOST --registry-config="${REGISTRY_CONFIG}" --push=gcr.io --tag=my-project/$APP-makisu:$CIRRUS_CHANGE_IN_REPO --file=./Dockerfile .
1.564305915197019e+09   info    Starting Makisu build (version=v0.1.11-10-g1b1102a)
1.5643059152263603e+09  info    Using build context: /tmp/cirrus-ci-build/resolve
1.564305915227681e+09   info    Using http server at http://127.0.0.1:12321 for cacheID storage
1.5643059154017467e+09  error   Failed to fetch intermediate layer with cache ID ec8aa8fa: find layer ec8aa8fa: layer not found in cache
1.5643059154018242e+09  info    * Stage 1/1 : (alias=0,latestfetched=-1)
1.56430591540184e+09    info    * Step 1/14 (commit,modifyfs) : FROM python:3.6.6  (446839ca)
1.5643059154020095e+09  info    * Started pulling image index.docker.io/library/python:3.6.6
1.5643059164474647e+09  info    * Started pulling layer index.docker.io/library/python:sha256:bc9ab73e5b14b9fbd3687a4d8c1f1360533d6ee9ffc3f5ecc6630794b40257b7
1.564305916644417e+09   info    * Started pulling layer index.docker.io/library/python:sha256:e5c3f8c317dc30af45021092a3d76f16ba7aa1ee5f18fec742c84d4960818580
1.5643059166470797e+09  info    * Started pulling layer index.docker.io/library/python:sha256:193a6306c92af328dbd41bbbd3200a2c90802624cccfe5725223324428110d7f
1.5643059170155766e+09  info    * Finished pulling layer library/python:e5c3f8c317dc30af45021092a3d76f16ba7aa1ee5f18fec742c84d4960818580
1.564305917079696e+09   info    * Started pulling layer index.docker.io/library/python:sha256:a587a86c9dcb9df6584180042becf21e36ecd8b460a761711227b4b06889a005
1.5643059171277025e+09  info    * Finished pulling layer library/python:193a6306c92af328dbd41bbbd3200a2c90802624cccfe5725223324428110d7f
1.5643059171895814e+09  info    * Finished pulling layer library/python:bc9ab73e5b14b9fbd3687a4d8c1f1360533d6ee9ffc3f5ecc6630794b40257b7
1.5643059171962411e+09  info    * Started pulling layer index.docker.io/library/python:sha256:72744d0a318b0788001cc4f5f83c6847ba4b753307fadd046b508bbc41eb9e29
1.5643059174475782e+09  info    * Started pulling layer index.docker.io/library/python:sha256:6598fc9d11d10365ac9281071a87930a2382ee31d026f1b6d432717b31db387c
1.5643059177758055e+09  info    * Finished pulling layer library/python:6598fc9d11d10365ac9281071a87930a2382ee31d026f1b6d432717b31db387c
1.5643059178413568e+09  info    * Started pulling layer index.docker.io/library/python:sha256:4b1d9004d467b4e710d770a881df027df7e5e7e4629f6e473760893ffc1a667f
1.5643059178802233e+09  info    * Finished pulling layer library/python:a587a86c9dcb9df6584180042becf21e36ecd8b460a761711227b4b06889a005
1.564305917945844e+09   info    * Started pulling layer index.docker.io/library/python:sha256:93612f47cdc374d0b33057b9e71eac173ac469da3e1a631dc8a32ba6986a408a
1.5643059183229275e+09  info    * Finished pulling layer library/python:93612f47cdc374d0b33057b9e71eac173ac469da3e1a631dc8a32ba6986a408a
1.5643059183920033e+09  info    * Started pulling layer index.docker.io/library/python:sha256:1bc4b4b508703799ef67a807dacce4736045e642e87bcd49871cd0f23e7f5b8b
1.564305918423843e+09   info    * Finished pulling layer library/python:4b1d9004d467b4e710d770a881df027df7e5e7e4629f6e473760893ffc1a667f
1.5643059184957218e+09  info    * Started pulling layer index.docker.io/library/python:sha256:8256ec07b2ad5dbbf324da48b498658f2fa47ea4ce859cb35d926c0234b486cb
1.564305918810201e+09   info    * Finished pulling layer library/python:1bc4b4b508703799ef67a807dacce4736045e642e87bcd49871cd0f23e7f5b8b
1.5643059188727944e+09  info    * Finished pulling layer library/python:8256ec07b2ad5dbbf324da48b498658f2fa47ea4ce859cb35d926c0234b486cb
1.5643059205010023e+09  info    * Finished pulling layer library/python:72744d0a318b0788001cc4f5f83c6847ba4b753307fadd046b508bbc41eb9e29
1.5643059205020914e+09  info    * Finished pulling image index.docker.io/library/python:3.6.6 in 5.100004363s
1.564305920506579e+09   info    * Processing FROM layer bc9ab73e5b14b9fbd3687a4d8c1f1360533d6ee9ffc3f5ecc6630794b40257b7
1.5643059219145367e+09  info    * Untarred 7001 files to / in 1.408s
1.564305921918238e+09   info    * Merged 7001 headers from tar to memfs
1.5643059219233994e+09  info    * Processing FROM layer 193a6306c92af328dbd41bbbd3200a2c90802624cccfe5725223324428110d7f
1.5643059222076356e+09  info    * Untarred 1304 files to / in 284ms
1.5643059222085664e+09  info    * Merged 1303 headers from tar to memfs
1.5643059222108777e+09  info    * Processing FROM layer e5c3f8c317dc30af45021092a3d76f16ba7aa1ee5f18fec742c84d4960818580
1.5643059223093393e+09  info    * Untarred 267 files to / in 98ms
1.564305922309646e+09   info    * Merged 262 headers from tar to memfs
1.564305922310478e+09   info    * Processing FROM layer a587a86c9dcb9df6584180042becf21e36ecd8b460a761711227b4b06889a005
1.5643059239247441e+09  info    * Untarred 6292 files to / in 1.614s
1.564305923928906e+09   info    * Merged 6288 headers from tar to memfs
1.5643059239316592e+09  info    * Processing FROM layer 72744d0a318b0788001cc4f5f83c6847ba4b753307fadd046b508bbc41eb9e29
1.5643059304517498e+09  info    * Untarred 20547 files to / in 6.52s
1.5643059304594743e+09  info    * Merged 20545 headers from tar to memfs
1.5643059304603536e+09  info    * Processing FROM layer 6598fc9d11d10365ac9281071a87930a2382ee31d026f1b6d432717b31db387c
1.5643059306866426e+09  info    * Untarred 672 files to / in 226ms
1.5643059306872816e+09  info    * Merged 667 headers from tar to memfs
1.5643059306881106e+09  info    * Processing FROM layer 4b1d9004d467b4e710d770a881df027df7e5e7e4629f6e473760893ffc1a667f
1.5643059312738402e+09  info    * Untarred 1104 files to / in 586ms
1.5643059312742963e+09  info    * Merged 1104 headers from tar to memfs
1.5643059312751646e+09  info    * Processing FROM layer 93612f47cdc374d0b33057b9e71eac173ac469da3e1a631dc8a32ba6986a408a
1.5643059312756255e+09  info    * Untarred 7 files to / in 0s
1.5643059312756627e+09  info    * Merged 7 headers from tar to memfs
1.5643059312765462e+09  info    * Processing FROM layer 1bc4b4b508703799ef67a807dacce4736045e642e87bcd49871cd0f23e7f5b8b
1.564305931374282e+09   info    * Untarred 596 files to / in 98ms
1.5643059313747387e+09  info    * Merged 596 headers from tar to memfs
1.5643059313747761e+09  info    * Execute FROM python:3.6.6  (446839ca) took 15.972880244s
1.5643059313752885e+09  info    * Step 2/14 (commit,modifyfs) : MAINTAINER John Doe <john@sdoe.com>  (ec8aa8fa)
1.564305931375339e+09   info    * Execute MAINTAINER John Doe <john@sdoe.com>  (ec8aa8fa) took 1.041µs
1.564305931375345e+09   info    * Pushing with cache ID ec8aa8fa
1.564305931375978e+09   info    * Step 3/14 (commit,modifyfs) : RUN mkdir -p /usr/src/resolve     && mkdir -p /usr/src/resolve/models     && pip install --upgrade pip  (944aa9ef)
1.564305932638471e+09   info    Collecting pip
1.5643059327380922e+09  info      Downloading https://files.pythonhosted.org/packages/62/ca/94d32a6516ed197a491d17d46595ce58a83cbb2fca280414e57cd86b84dc/pip-19.2.1-py2.py3-none-any.whl (1.4MB)
1.564305932872864e+09   info    Installing collected packages: pip
1.5643059328729243e+09  info      Found existing installation: pip 18.1
1.5643059329885137e+09  info        Uninstalling pip-18.1:
1.5643059331684113e+09  info          Successfully uninstalled pip-18.1
1.564305933981709e+09   info    Successfully installed pip-19.2.1
1.5643059341407273e+09  info    * Execute RUN mkdir -p /usr/src/resolve     && mkdir -p /usr/src/resolve/models     && pip install --upgrade pip  (944aa9ef) took 2.76471766s
1.5643059390795004e+09  info    * Collecting filesystem diff
1.564305939913849e+09   info    * Finished collecting diff in 834ms: 1025 files found
1.5643059401578593e+09  info    * Created layer by scanning filesystem; 1025 files found
1.5643059401671553e+09  info    * Committed gzipped layer sha256:d3fc069215c7ba6059cf21b0b463c00c996710b66fe4fee5339b20b83b6778d0 (8472575 bytes)
1.564305940167195e+09   info    * Pushing with cache ID 944aa9ef
1.5643059401678967e+09  info    * Step 4/14 (commit,modifyfs) : RUN pip install gunicorn==19.9.0  (1d7c7f3f)
1.5643059405633643e+09  info    Collecting gunicorn==19.9.0
1.5643059407111268e+09  info      Downloading https://files.pythonhosted.org/packages/8c/da/b8dd8deb741bff556db53902d4706774c8e1e67265f69528c14c003644e6/gunicorn-19.9.0-py2.py3-none-any.whl (112kB)
1.5643059407638924e+09  info    Installing collected packages: gunicorn
1.564305940856079e+09   info    Successfully installed gunicorn-19.9.0
1.5643059409177616e+09  info    * Execute RUN pip install gunicorn==19.9.0  (1d7c7f3f) took 749.802494ms
1.564305941918179e+09   info    * Collecting filesystem diff
1.5643059426625617e+09  info    * Finished collecting diff in 744ms: 126 files found
1.5643059426728718e+09  info    * Created layer by scanning filesystem; 126 files found
1.5643059426792467e+09  info    * Committed gzipped layer sha256:34d73d444672f815723843b77fbb692893dc1e67fa779eb692b029994157ae74 (410075 bytes)
1.564305942679564e+09   info    * Pushing with cache ID 1d7c7f3f
1.5643059426803443e+09  info    * Step 5/14 (commit,modifyfs) : RUN pip install poetry  (d3273579)
1.564305943067599e+09   info    Collecting poetry
1.5643059432930908e+09  info      Downloading https://files.pythonhosted.org/packages/c9/76/214acc400055aa0045ed21ce25378225b1887c5c091f99476955bf9f6984/poetry-0.12.17-py2.py3-none-any.whl (195kB)
1.5643059433920708e+09  info    Collecting pyrsistent<0.15.0,>=0.14.2 (from poetry)
1.564305943430971e+09   info      Downloading https://files.pythonhosted.org/packages/8c/46/4e93ab8a379d7efe93f20a0fb8a27bdfe88942cc954ab0210c3164e783e0/pyrsistent-0.14.11.tar.gz (104kB)
1.5643059438448305e+09  info    Collecting pyparsing<3.0,>=2.2 (from poetry)
1.5643059439204583e+09  info      Downloading https://files.pythonhosted.org/packages/ff/2c/afc36631a9e0dc9f2b8bd4c19fa2d330cbe07d6679a1c0910418a200acad/pyparsing-2.4.1.1-py2.py3-none-any.whl (64kB)
1.5643059439336166e+09  info    Collecting pkginfo<2.0,>=1.4 (from poetry)
1.5643059440364084e+09  info      Downloading https://files.pythonhosted.org/packages/e6/d5/451b913307b478c49eb29084916639dc53a88489b993530fed0a66bab8b9/pkginfo-1.5.0.1-py2.py3-none-any.whl
1.5643059440526583e+09  info    Collecting requests<3.0,>=2.18 (from poetry)
1.5643059441294332e+09  info      Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)
1.5643059441554132e+09  info    Collecting cachecontrol[filecache]<0.13.0,>=0.12.4 (from poetry)
1.5643059441905816e+09  info      Downloading https://files.pythonhosted.org/packages/5e/f0/2c193ed1f17c97ae539da7e1c2d48b80d8cccb1917163b26a91ca4355aa6/CacheControl-0.12.5.tar.gz
1.56430594443543e+09    info    Collecting shellingham<2.0,>=1.1 (from poetry)
1.5643059446227052e+09  info      Downloading https://files.pythonhosted.org/packages/2d/ac/e8a34d4b3d24bf554f40651b2aac549a3fc7223725bf10fbdfe2083b6372/shellingham-1.3.1-py2.py3-none-any.whl
1.5643059446344335e+09  info    Collecting tomlkit<0.6.0,>=0.5.1 (from poetry)
1.564305944745968e+09   info      Downloading https://files.pythonhosted.org/packages/29/22/716eb55bb154d2519752a2d91cf7e91d58dd24e8150c47aaaa67aae75aa6/tomlkit-0.5.5-py2.py3-none-any.whl
1.5643059447651706e+09  info    Collecting cachy<0.3,>=0.2 (from poetry)
1.5643059447942119e+09  info      Downloading https://files.pythonhosted.org/packages/26/37/8ce3e7b330078b6797a34e79a80a8ad6935e404a3b903765417182c9ce19/cachy-0.2.0-py2.py3-none-any.whl (59kB)
1.564305944810941e+09   info    Collecting cleo<0.7.0,>=0.6.7 (from poetry)
1.564305944854549e+09   info      Downloading https://files.pythonhosted.org/packages/a7/b9/270301a3a87587f09bc3985973f2e362ffa45fa5fcd5128501516b2f5e31/cleo-0.6.8-py2.py3-none-any.whl (264kB)
1.5643059448843765e+09  info    Collecting jsonschema<4.0,>=3.0a3 (from poetry)
1.5643059449256701e+09  info      Downloading https://files.pythonhosted.org/packages/aa/69/df679dfbdd051568b53c38ec8152a3ab6bc533434fc7ed11ab034bf5e82f/jsonschema-3.0.1-py2.py3-none-any.whl (54kB)
1.5643059449496784e+09  info    Collecting html5lib<2.0,>=1.0 (from poetry)
1.5643059449796767e+09  info      Downloading https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl (117kB)
1.564305945010845e+09   info    Collecting requests-toolbelt<0.9.0,>=0.8.0 (from poetry)
1.564305945043501e+09   info      Downloading https://files.pythonhosted.org/packages/97/8a/d710f792d6f6ecc089c5e55b66e66c3f2f35516a1ede5a8f54c13350ffb0/requests_toolbelt-0.8.0-py2.py3-none-any.whl (54kB)
1.5643059450601454e+09  info    Collecting six (from pyrsistent<0.15.0,>=0.14.2->poetry)
1.564305945094142e+09   info      Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
1.5643059451044638e+09  info    Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests<3.0,>=2.18->poetry)
1.564305945157288e+09   info      Downloading https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
1.564305945190462e+09   info    Collecting chardet<3.1.0,>=3.0.2 (from requests<3.0,>=2.18->poetry)
1.564305945220159e+09   info      Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
1.564305945239445e+09   info    Collecting idna<2.9,>=2.5 (from requests<3.0,>=2.18->poetry)
1.5643059452718954e+09  info      Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
1.5643059452844913e+09  info    Collecting certifi>=2017.4.17 (from requests<3.0,>=2.18->poetry)
1.5643059453219824e+09  info      Downloading https://files.pythonhosted.org/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl (157kB)
1.564305945335415e+09   info    Collecting msgpack (from cachecontrol[filecache]<0.13.0,>=0.12.4->poetry)
1.5643059453841977e+09  info      Downloading https://files.pythonhosted.org/packages/92/7e/ae9e91c1bb8d846efafd1f353476e3fd7309778b582d2fb4cea4cc15b9a2/msgpack-0.6.1-cp36-cp36m-manylinux1_x86_64.whl (248kB)
1.5643059454019456e+09  info    Collecting lockfile>=0.9 (from cachecontrol[filecache]<0.13.0,>=0.12.4->poetry)
1.5643059454295585e+09  info      Downloading https://files.pythonhosted.org/packages/c8/22/9460e311f340cb62d26a38c419b1381b8593b0bb6b5d1f056938b086d362/lockfile-0.12.2-py2.py3-none-any.whl
1.5643059454398437e+09  info    Collecting pylev<2.0,>=1.3 (from cleo<0.7.0,>=0.6.7->poetry)
1.5643059454654098e+09  info      Downloading https://files.pythonhosted.org/packages/40/1c/7dff1d242bf1e19f9c6202f0ba4e6fd18cc7ecb8bc85b17b2d16c806e228/pylev-1.3.0-py2.py3-none-any.whl
1.564305945475453e+09   info    Collecting pastel<0.2.0,>=0.1.0 (from cleo<0.7.0,>=0.6.7->poetry)
1.564305945500086e+09   info      Downloading https://files.pythonhosted.org/packages/9b/7e/7d701686013c0d7dae62e0977467232a6adc2e562c23878eb3cd4f97d02e/pastel-0.1.0-py3-none-any.whl
1.564305945508103e+09   info    Requirement already satisfied: setuptools in /usr/local/lib/python3.6/site-packages (from jsonschema<4.0,>=3.0a3->poetry) (40.4.3)
1.5643059455151021e+09  info    Collecting attrs>=17.4.0 (from jsonschema<4.0,>=3.0a3->poetry)
1.5643059455465586e+09  info      Downloading https://files.pythonhosted.org/packages/23/96/d828354fa2dbdf216eaa7b7de0db692f12c234f7ef888cc14980ef40d1d2/attrs-19.1.0-py2.py3-none-any.whl
1.5643059455785832e+09  info    Collecting webencodings (from html5lib<2.0,>=1.0->poetry)
1.5643059456049016e+09  info      Downloading https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl
1.5643059456131427e+09  info    Building wheels for collected packages: pyrsistent, cachecontrol
1.564305945613847e+09   info      Building wheel for pyrsistent (setup.py): started
1.5643059467257838e+09  info      Building wheel for pyrsistent (setup.py): finished with status 'done'
1.5643059467273457e+09  info      Created wheel for pyrsistent: filename=pyrsistent-0.14.11-cp36-cp36m-linux_x86_64.whl size=111967 sha256=da8666f6712c632306b3c20850594257afb26a7efb1c5469c8946e28229c9b5c
1.56430594672744e+09    info      Stored in directory: /root/.cache/pip/wheels/83/59/9a/a037b9b3c3e93d9275ea0aff9d6064400f372879dfdab01afe
1.5643059467409644e+09  info      Building wheel for cachecontrol (setup.py): started
1.564305947032157e+09   info      Building wheel for cachecontrol (setup.py): finished with status 'done'
1.5643059470333097e+09  info      Created wheel for cachecontrol: filename=CacheControl-0.12.5-cp36-none-any.whl size=18489 sha256=e90903deaf4e4baaa5895e02472a735e58449736e0fdb8a95da10bb5ebb8e371
1.564305947033484e+09   info      Stored in directory: /root/.cache/pip/wheels/36/bd/5f/dbbee4f2d51f97ecd12a363f870361179cb1fd4bc1174ea08a
1.564305947039347e+09   info    Successfully built pyrsistent cachecontrol
1.5643059471697376e+09  info    Installing collected packages: six, pyrsistent, pyparsing, pkginfo, urllib3, chardet, idna, certifi, requests, msgpack, lockfile, cachecontrol, shellingham, tomlkit, cachy, pylev, pastel, cleo, attrs, jsonschema, webencodings, html5lib, requests-toolbelt, poetry
1.5643059482160053e+09  info    Successfully installed attrs-19.1.0 cachecontrol-0.12.5 cachy-0.2.0 certifi-2019.6.16 chardet-3.0.4 cleo-0.6.8 html5lib-1.0.1 idna-2.8 jsonschema-3.0.1 lockfile-0.12.2 msgpack-0.6.1 pastel-0.1.0 pkginfo-1.5.0.1 poetry-0.12.17 pylev-1.3.0 pyparsing-2.4.1.1 pyrsistent-0.14.11 requests-2.22.0 requests-toolbelt-0.8.0 shellingham-1.3.1 six-1.12.0 tomlkit-0.5.5 urllib3-1.25.3 webencodings-0.5.1
1.5643059483234994e+09  info    * Execute RUN pip install poetry  (d3273579) took 5.643102222s
1.5643059493240607e+09  info    * Collecting filesystem diff
1.564305950197831e+09   info    * Finished collecting diff in 874ms: 1910 files found
1.5643059503661907e+09  info    * Created layer by scanning filesystem; 1910 files found
1.5643059503718877e+09  info    * Committed gzipped layer sha256:6f7445cbad659141bfb874ad19a9704bd25b43417e469e7ba697fda138bc8009 (5248391 bytes)
1.5643059503719156e+09  info    * Pushing with cache ID d3273579
1.5643059503725297e+09  info    * Step 6/14 (commit,modifyfs) : WORKDIR /usr/src/resolve  (d566191)
1.5643059503725784e+09  info    * Execute WORKDIR /usr/src/resolve  (d566191) took 1.042µs
1.5643059503725832e+09  info    * Pushing with cache ID d566191
1.5643059503730712e+09  info    * Step 7/14 (commit,modifyfs) : COPY secrets secrets  (42e75608)
1.564305950373489e+09   info    * Execute COPY secrets secrets  (42e75608) took 387.989µs
1.564305951374436e+09   info    * Created copy layer with 6 files
1.5643059513761637e+09  info    * Committed gzipped layer sha256:8a041c71e7ccc1418342afce952b8115ea1351db58f128c06b7beece6ebe561f (2124 bytes)
1.56430595137652e+09    info    * Pushing with cache ID 42e75608
1.5643059513776245e+09  info    * Step 8/14 (commit,modifyfs) : COPY pyproject.toml poetry.lock install_dependencies.sh /usr/src/resolve/  (c891b855)
1.5643059513787923e+09  info    * Execute COPY pyproject.toml poetry.lock install_dependencies.sh /usr/src/resolve/  (c891b855) took 825.145µs
1.5643059523801625e+09  info    * Created copy layer with 6 files
1.5643059523830843e+09  info    * Committed gzipped layer sha256:b8a07cfb875b90ae3fd5d667dd651b0feba0d5956836b30e9ed9fb262fd68832 (22301 bytes)
1.564305952383427e+09   info    * Pushing with cache ID c891b855
1.5643059523842015e+09  info    * Step 9/14 (commit,modifyfs) : ARG GEMFURY_TOKEN  (5d030392)
1.5643059523844893e+09  info    * Execute ARG GEMFURY_TOKEN  (5d030392) took 5.684µs
1.5643059523847103e+09  info    * Pushing with cache ID 5d030392
1.56430595238546e+09    info    * Step 10/14 (commit,modifyfs) : RUN GEMFURY_TOKEN=${GEMFURY_TOKEN} ./install_dependencies.sh  (c1bf0f49)
1.564305953377296e+09   info    Skipping virtualenv creation, as specified in config file.
1.564305954158073e+09   info    Installing dependencies from lock file
1.564305954832293e+09   info    
1.5643059548327742e+09  info    
Package operations: 68 installs, 5 updates, 0 removals
  - Installing pyasn1 (0.4.5)
1.5643059555791357e+09  info      - Updating six (1.12.0 -> 1.11.0)
1.5643059561935637e+09  info      - Installing cachetools (3.1.1)
1.5643059568062196e+09  info      - Updating idna (2.8 -> 2.7)
1.564305957485989e+09   info      - Installing protobuf (3.9.0)
1.5643059587350888e+09  info      - Installing pyasn1-modules (0.2.5)
1.5643059595340626e+09  info      - Installing rsa (4.0)
1.5643059602112775e+09  info      - Updating urllib3 (1.25.3 -> 1.23)
1.5643059609435349e+09  info      - Installing docutils (0.14)
1.5643059619535859e+09  info      - Installing google-auth (1.6.3)
1.564305962628653e+09   info      - Installing googleapis-common-protos (1.6.0)
1.5643059639617796e+09  info      - Installing jmespath (0.9.4)
1.5643059645882602e+09  info      - Installing python-dateutil (2.8.0)
1.5643059652957408e+09  info      - Installing pytz (2019.1)
1.5643059663173704e+09  info      - Updating requests (2.22.0 -> 2.19.1)
1.564305967041387e+09   info      - Installing botocore (1.12.191)
1.564305969096692e+09   info      - Installing google-api-core (1.14.0)
1.5643059697952476e+09  info      - Installing appdirs (1.4.3)
1.5643059704192889e+09  info      - Installing cymem (2.0.2)
1.5643059710335398e+09  info      - Installing google-cloud-core (0.29.1)
1.5643059716883433e+09  info      - Installing google-resumable-media (0.3.2)
1.5643059723538845e+09  info      - Updating msgpack (0.6.1 -> 0.5.6)
1.5643059730519335e+09  info      - Installing numpy (1.15.0)
1.5643059760313275e+09  info      - Installing s3transfer (0.2.1)
1.5643059767053251e+09  info      - Installing toolz (0.10.0)
1.5643059781025777e+09  info      - Installing boto (2.49.0)
1.5643059795698204e+09  info      - Installing boto3 (1.9.191)
1.5643059805252273e+09  info      - Installing cytoolz (0.9.0.1)
1.5643060000662808e+09  info      - Installing dill (0.2.8.2)
1.5643060013874466e+09  info      - Installing fs (2.4.8)
1.564306002206525e+09   info      - Installing google-cloud-storage (1.13.2)
1.564306002910627e+09   info      - Installing marisa-trie (0.7.5)
1.564306019017573e+09   info      - Installing markupsafe (1.1.1)
1.564306019688849e+09   info      - Installing msgpack-numpy (0.4.3.2)
1.5643060203552153e+09  info      - Installing murmurhash (1.0.2)
1.564306021031228e+09   info      - Installing plac (0.9.6)
1.5643060217113945e+09  info      - Installing preshed (2.0.1)
1.5643060224288566e+09  info      - Installing tqdm (4.24.0)
1.5643060231660595e+09  info      - Installing wrapt (1.10.11)
1.5643060250027034e+09  info      - Installing click (7.0)
1.5643060257711194e+09  info      - Installing enum34 (1.1.6)
1.5643060264251077e+09  info      - Installing fs-gcsfs (0.4.1)
1.564306027680222e+09   info      - Installing futures (3.1.1)
1.564306028963687e+09   info      - Installing itsdangerous (1.1.0)
1.5643060296585975e+09  info      - Installing jinja2 (2.10.1)
1.5643060304462152e+09  info      - Installing langcodes (1.4.1)
1.5643060327761793e+09  info      - Installing psutil (5.6.3)
1.564306035432153e+09   info      - Installing regex (2018.01.10)
1.5643060463252249e+09  info      - Installing scipy (1.3.0)
1.56430605047923e+09    info      - Installing smart-open (1.8.4)
1.5643060518452435e+09  info      - Installing thinc (6.12.1)
1.5643060528789866e+09  info      - Installing ujson (1.35)
1.5643060551091783e+09  info      - Installing websocket-client (0.56.0)
1.5643060558496518e+09  info      - Installing werkzeug (0.15.5)
1.5643060567324436e+09  info      - Installing beautifulsoup4 (4.6.3)
1.56430605749315e+09    info      - Installing ddtrace (0.27.0)
1.5643060601991947e+09  info      - Installing flask (1.0.2)
1.5643060609589949e+09  info      - Installing gensim (3.6.0)
1.564306062983306e+09   info      - Installing graphqlclient (0.2.4)
1.564306064265957e+09   info      - Installing grpcio (1.18.0)
1.56430606597738e+09    info      - Installing janome (0.3.6)
1.5643060971549213e+09  info      - Installing jieba (0.39)
1.5643061017100897e+09  info      - Installing joblib (0.11)
1.5643061025740614e+09  info      - Installing more-itertools (4.1.0)
1.5643061033437889e+09  info      - Installing nltk (3.3)
1.5643061062801018e+09  info      - Installing pandas (0.21.1)
1.5643061110482614e+09  info      - Installing pika (0.10.0)
1.564306111845532e+09   info      - Installing pysolvvy (2.2.1)
1.5643061140786307e+09  info      - Installing python-json-logger (0.1.10)
1.5643061147993839e+09  info      - Installing scikit-learn (0.20.2)
1.5643061171768298e+09  info      - Installing slackclient (1.1.0)
1.5643061185465374e+09  info      - Installing spacy (2.0.18)
1.564306158019171e+09   info      - Installing wordfreq (2.2.1)
1.5643061595072303e+09  error   /usr/local/lib/python3.6/site-packages/poetry/version/requirements.py:170: UserWarning: warn_ungrouped_named_tokens_in_collection: setting results name 'specifier' on And expression collides with '_original_end' on contained expression
  VERSION_SPEC = originalTextFor(_VERSION_SPEC)("specifier")
/usr/local/lib/python3.6/site-packages/poetry/version/requirements.py:173: UserWarning: warn_ungrouped_named_tokens_in_collection: setting results name 'marker' on And expression collides with '_original_end' on contained expression
  MARKER_EXPR = originalTextFor(MARKER_EXPR())("marker")
1.5643061608907151e+09  info    [nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data] Error downloading 'stopwords' from
[nltk_data]     <https://raw.githubusercontent.com/nltk/nltk_data/gh-
[nltk_data]     pages/packages/corpora/stopwords.zip>:   <urlopen
[nltk_data]     error [SSL: CERTIFICATE_VERIFY_FAILED] certificate
[nltk_data]     verify failed (_ssl.c:841)>
[nltk_data] Downloading package wordnet to /root/nltk_data...
[nltk_data] Error downloading 'wordnet' from
[nltk_data]     <https://raw.githubusercontent.com/nltk/nltk_data/gh-
[nltk_data]     pages/packages/corpora/wordnet.zip>:   <urlopen error
[nltk_data]     [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
[nltk_data]     failed (_ssl.c:841)>
1.564306162135779e+09   info    Collecting en_core_web_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz#egg=en_core_web_sm==2.0.0
1.5643061626807895e+09  info      Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz (37.4MB)
1.564306164976116e+09   info    Building wheels for collected packages: en-core-web-sm
1.5643061649767957e+09  info      Building wheel for en-core-web-sm (setup.py): started
1.5643061674775896e+09  info      Building wheel for en-core-web-sm (setup.py): finished with status 'done'
1.5643061675932443e+09  info      Created wheel for en-core-web-sm: filename=en_core_web_sm-2.0.0-cp36-none-any.whl size=37405977 sha256=fc6520997c2d51144ce8d565b6bbe502aef2781568d460b3ef8b36928eedd787
  Stored in directory: /tmp/pip-ephem-wheel-cache-jvhe75oa/wheels/54/7c/d8/f86364af8fbba7258e14adae115f18dd2c91552406edc3fdaa
1.5643061679790123e+09  info    Successfully built en-core-web-sm
1.564306167979127e+09   info    Installing collected packages: en-core-web-sm
1.5643061680692036e+09  info    Successfully installed en-core-web-sm-2.0.0
1.5643061682762437e+09  info    
    Linking successful
    /usr/local/lib/python3.6/site-packages/en_core_web_sm -->
    /usr/local/lib/python3.6/site-packages/spacy/data/en
    You can now load the model via spacy.load('en')
1.5643061693969872e+09  info    Collecting de_core_news_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-2.0.0/de_core_news_sm-2.0.0.tar.gz#egg=de_core_news_sm==2.0.0
1.56430616987158e+09    info      Downloading https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-2.0.0/de_core_news_sm-2.0.0.tar.gz (38.2MB)
1.5643061722450833e+09  info    Building wheels for collected packages: de-core-news-sm
1.5643061722457318e+09  info      Building wheel for de-core-news-sm (setup.py): started
1.5643061749243102e+09  info      Building wheel for de-core-news-sm (setup.py): finished with status 'done'
1.5643061750428846e+09  info      Created wheel for de-core-news-sm: filename=de_core_news_sm-2.0.0-cp36-none-any.whl size=38276259 sha256=e0690a31f47667c99a814fa47248ea4bdc3e37bd4402bdf99ff1ff5bd38cb5eb
1.5643061750429406e+09  info      Stored in directory: /tmp/pip-ephem-wheel-cache-reyyky7j/wheels/a5/9e/96/785f498e5cea276b337aab8a79259d32a93f03bd744d71eef5
1.564306175453096e+09   info    Successfully built de-core-news-sm
1.564306175453699e+09   info    Installing collected packages: de-core-news-sm
1.564306175546654e+09   info    Successfully installed de-core-news-sm-2.0.0
1.5643061757399292e+09  info    
    Linking successful
    /usr/local/lib/python3.6/site-packages/de_core_news_sm -->
    /usr/local/lib/python3.6/site-packages/spacy/data/de
    You can now load the model via spacy.load('de')
1.5643061768918226e+09  info    Collecting es_core_news_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/es_core_news_sm-2.0.0/es_core_news_sm-2.0.0.tar.gz#egg=es_core_news_sm==2.0.0
1.5643061774281178e+09  info      Downloading https://github.com/explosion/spacy-models/releases/download/es_core_news_sm-2.0.0/es_core_news_sm-2.0.0.tar.gz (36.7MB)
1.5643061796637475e+09  info    Building wheels for collected packages: es-core-news-sm
1.5643061796643817e+09  info      Building wheel for es-core-news-sm (setup.py): started
1.5643061820971627e+09  info      Building wheel for es-core-news-sm (setup.py): finished with status 'done'
1.5643061822095928e+09  info      Created wheel for es-core-news-sm: filename=es_core_news_sm-2.0.0-cp36-none-any.whl size=36749073 sha256=11c8bf4a62ab78328e3ad43d2103d97e96658fa72e6005ac07784d7e2ed928fa
1.5643061822096481e+09  info      Stored in directory: /tmp/pip-ephem-wheel-cache-bdvi887k/wheels/9e/28/c4/df4980946eb229379ed26d349566e427fa029dbf03546ccb94
1.564306182585842e+09   info    Successfully built es-core-news-sm
1.5643061825859742e+09  info    Installing collected packages: es-core-news-sm
1.564306182672747e+09   info    Successfully installed es-core-news-sm-2.0.0
1.5643061828758357e+09  info    
    Linking successful
    /usr/local/lib/python3.6/site-packages/es_core_news_sm -->
    /usr/local/lib/python3.6/site-packages/spacy/data/es
    You can now load the model via spacy.load('es')
1.5643061840248098e+09  info    Collecting pt_core_news_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/pt_core_news_sm-2.0.0/pt_core_news_sm-2.0.0.tar.gz#egg=pt_core_news_sm==2.0.0
1.5643061845828686e+09  info      Downloading https://github.com/explosion/spacy-models/releases/download/pt_core_news_sm-2.0.0/pt_core_news_sm-2.0.0.tar.gz (38.7MB)
1.5643061870466504e+09  info    Building wheels for collected packages: pt-core-news-sm
1.5643061870473793e+09  info      Building wheel for pt-core-news-sm (setup.py): started
1.564306189780281e+09   info      Building wheel for pt-core-news-sm (setup.py): finished with status 'done'
1.564306189899478e+09   info      Created wheel for pt-core-news-sm: filename=pt_core_news_sm-2.0.0-cp36-none-any.whl size=38749345 sha256=52a1e61705a91ade7b5e579dffbfb5d9ee6b78f6913d3af460c585bf08481d30
1.5643061898995314e+09  info      Stored in directory: /tmp/pip-ephem-wheel-cache-k4_3ntbs/wheels/0b/f1/54/a8f759b41cf39d2a33d6da84cd90b8e2fefea549963396bbd2
1.5643061902959418e+09  info    Successfully built pt-core-news-sm
1.5643061902959847e+09  info    Installing collected packages: pt-core-news-sm
1.5643061903886478e+09  info    Successfully installed pt-core-news-sm-2.0.0
1.5643061905912204e+09  info    
    Linking successful
    /usr/local/lib/python3.6/site-packages/pt_core_news_sm -->
    /usr/local/lib/python3.6/site-packages/spacy/data/pt
    You can now load the model via spacy.load('pt')
1.5643061917029674e+09  info    Collecting fr_core_news_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/fr_core_news_sm-2.0.0/fr_core_news_sm-2.0.0.tar.gz#egg=fr_core_news_sm==2.0.0
1.56430619242016e+09    info      Downloading https://github.com/explosion/spacy-models/releases/download/fr_core_news_sm-2.0.0/fr_core_news_sm-2.0.0.tar.gz (39.8MB)
1.5643061949027922e+09  info    Building wheels for collected packages: fr-core-news-sm
1.564306194903339e+09   info      Building wheel for fr-core-news-sm (setup.py): started
1.5643061977933373e+09  info      Building wheel for fr-core-news-sm (setup.py): finished with status 'done'
1.56430619791312e+09    info      Created wheel for fr-core-news-sm: filename=fr_core_news_sm-2.0.0-cp36-none-any.whl size=39841461 sha256=29a168848439ab9c70e303b113f7bc6cf2c749a0d5502d0e4a220086e1ccdd69
  Stored in directory: /tmp/pip-ephem-wheel-cache-1ffxtfgb/wheels/50/cb/17/4b4c5a59786e77f375913b22b395922d768b069b7d7aa7bd24
1.5643061983669324e+09  info    Successfully built fr-core-news-sm
1.564306198366972e+09   info    Installing collected packages: fr-core-news-sm
1.5643061984732761e+09  info    Successfully installed fr-core-news-sm-2.0.0
1.5643061986964142e+09  info    
    Linking successful
    /usr/local/lib/python3.6/site-packages/fr_core_news_sm -->
    /usr/local/lib/python3.6/site-packages/spacy/data/fr
    You can now load the model via spacy.load('fr')
1.5643061998371525e+09  info    Collecting it_core_news_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/it_core_news_sm-2.0.0/it_core_news_sm-2.0.0.tar.gz#egg=it_core_news_sm==2.0.0
1.5643062004175065e+09  info      Downloading https://github.com/explosion/spacy-models/releases/download/it_core_news_sm-2.0.0/it_core_news_sm-2.0.0.tar.gz (36.5MB)
1.5643062027487109e+09  info    Building wheels for collected packages: it-core-news-sm
1.5643062027492805e+09  info      Building wheel for it-core-news-sm (setup.py): started
1.5643062051313343e+09  info      Building wheel for it-core-news-sm (setup.py): finished with status 'done'
1.564306205242413e+09   info      Created wheel for it-core-news-sm: filename=it_core_news_sm-2.0.0-cp36-none-any.whl size=36476091 sha256=c0ffa25968f429287832156371507b72791c4c37d88d7005ae56c056f2440359
1.5643062052424557e+09  info      Stored in directory: /tmp/pip-ephem-wheel-cache-zt774awq/wheels/c4/cb/1c/c452364dfe51ffb6ab2727df879e833565d27e255c76ae2954
1.5643062056123633e+09  info    Successfully built it-core-news-sm
1.564306205612407e+09   info    Installing collected packages: it-core-news-sm
1.56430620570114e+09    info    Successfully installed it-core-news-sm-2.0.0
1.5643062059075215e+09  info    
    Linking successful
    /usr/local/lib/python3.6/site-packages/it_core_news_sm -->
    /usr/local/lib/python3.6/site-packages/spacy/data/it
    You can now load the model via spacy.load('it')
1.5643062070487492e+09  info    Collecting nl_core_news_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/nl_core_news_sm-2.0.0/nl_core_news_sm-2.0.0.tar.gz#egg=nl_core_news_sm==2.0.0
1.5643062078050048e+09  info      Downloading https://github.com/explosion/spacy-models/releases/download/nl_core_news_sm-2.0.0/nl_core_news_sm-2.0.0.tar.gz (36.7MB)
1.5643062101088996e+09  info    Building wheels for collected packages: nl-core-news-sm
1.56430621010946e+09    info      Building wheel for nl-core-news-sm (setup.py): started
1.564306212561634e+09   info      Building wheel for nl-core-news-sm (setup.py): finished with status 'done'
1.5643062126733632e+09  info      Created wheel for nl-core-news-sm: filename=nl_core_news_sm-2.0.0-cp36-none-any.whl size=36694098 sha256=169d6a1c470c7e4c8ebea3e949a81baff3d08c6d9931a2a2f0c12acb1484c498
1.5643062126734076e+09  info      Stored in directory: /tmp/pip-ephem-wheel-cache-ux6ocaaf/wheels/20/e1/45/57eaecebfacb4fd1dcb8cc4e1ef0f71e4b8d72043b0ec8f1e5
1.564306213042146e+09   info    Successfully built nl-core-news-sm
1.564306213042793e+09   info    Installing collected packages: nl-core-news-sm
1.564306213130649e+09   info    Successfully installed nl-core-news-sm-2.0.0
1.5643062133329954e+09  info    
    Linking successful
    /usr/local/lib/python3.6/site-packages/nl_core_news_sm -->
    /usr/local/lib/python3.6/site-packages/spacy/data/nl
    You can now load the model via spacy.load('nl')
1.5643062139693282e+09  info    
    Linking successful
    /usr/local/lib/python3.6/site-packages/jieba -->
    /usr/local/lib/python3.6/site-packages/spacy/data/zh
    You can now load the model via spacy.load('zh')
1.5643062145512326e+09  info    
    Linking successful
    /usr/local/lib/python3.6/site-packages/janome -->
    /usr/local/lib/python3.6/site-packages/spacy/data/ja
    You can now load the model via spacy.load('ja')
1.5643062145969667e+09  info    * Execute RUN GEMFURY_TOKEN=${GEMFURY_TOKEN} ./install_dependencies.sh  (c1bf0f49) took 4m22.211239423s
1.5643062155974643e+09  info    * Collecting filesystem diff
1.5643062184818888e+09  info    * Finished collecting diff in 2.884s: 16726 files found
1.564306231832618e+09   info    * Created layer by scanning filesystem; 16726 files found
1.5643062318390176e+09  info    * Committed gzipped layer sha256:e7d8cecf15ca1d353859cb01d3dc981d83c2688f67c0c28e63a4a5059b6e7d7b (822886598 bytes)
1.564306231839078e+09   info    * Pushing with cache ID c1bf0f49
1.5643062318397386e+09  info    * Step 11/14 (commit,modifyfs) : COPY . /usr/src/resolve  (bb7e73c6)
1.564306231853637e+09   info    * Execute COPY . /usr/src/resolve  (bb7e73c6) took 13.845266ms
1.5643062350982347e+09  info    * Created copy layer with 115 files
1.5643062351020508e+09  info    * Committed gzipped layer sha256:5a48e99eaaff29fe6775cd72fb0457da90977cbdb3a66acaafa5a9dcd4723d2a (141533 bytes)
1.5643062351020858e+09  info    * Pushing with cache ID bb7e73c6
1.5643062351027906e+09  info    * Step 12/14 (commit,modifyfs) : ENV GOOGLE_APPLICATION_CREDENTIALSS=/usr/src/resolve/secrets/my-project-key.json     PYTHONUNBUFFERED=TRUE     PYTHONDONTWRITEBYTECODE=TRUE     PYTHONHASHSEED=0     PYTHONPATH="/usr/src/resolve/src"  (20ed4a5c)
1.564306235102849e+09   info    * Execute ENV GOOGLE_APPLICATION_CREDENTIALSS=/usr/src/resolve/secrets/my-project-key.json     PYTHONUNBUFFERED=TRUE     PYTHONDONTWRITEBYTECODE=TRUE     PYTHONHASHSEED=0     PYTHONPATH="/usr/src/resolve/src"  (20ed4a5c) took 1.619µs
1.564306235102856e+09   info    * Pushing with cache ID 20ed4a5c
1.5643062351034133e+09  info    * Step 13/14 (commit,modifyfs) : EXPOSE 6062  (b35c2a2a)
1.5643062351034636e+09  info    * Execute EXPOSE 6062  (b35c2a2a) took 1.35µs
1.5643062351034722e+09  info    * Pushing with cache ID b35c2a2a
1.5643062351040144e+09  info    * Step 14/14 (commit,modifyfs) : ENTRYPOINT ddtrace-run gunicorn --name=resolve_api  --log-config=src/config/gunicorn_logging.conf --config=src/config/gunicorn_config.py --access-logformat=\"{\'remote_ip\':\'%\(h\)s\'\,\'code\':\'%\(s\)s\'\,\'method\':\'%\(m\)s\'\,\'path\':\'%\(U\)s\'\,\'querystring\':\'%\(q\)s\'\,\'timetaken\':\'%\(D\)s\'\,\'response_length\':\'%\(B\)s\'}\" resolve_api:configure_app\(remote_model_dir=\"$REMOTE_MODEL_DIR\"\,initial_model_version=\"$INITIAL_MODEL_VERSION\"\,local_model_dir=\"/data\"\,org_id=$ORG_ID\)  (eb427ee8)
1.564306235104085e+09   info    * Execute ENTRYPOINT ddtrace-run gunicorn --name=resolve_api  --log-config=src/config/gunicorn_logging.conf --config=src/config/gunicorn_config.py --access-logformat=\"{\'remote_ip\':\'%\(h\)s\'\,\'code\':\'%\(s\)s\'\,\'method\':\'%\(m\)s\'\,\'path\':\'%\(U\)s\'\,\'querystring\':\'%\(q\)s\'\,\'timetaken\':\'%\(D\)s\'\,\'response_length\':\'%\(B\)s\'}\" resolve_api:configure_app\(remote_model_dir=\"$REMOTE_MODEL_DIR\"\,initial_model_version=\"$INITIAL_MODEL_VERSION\"\,local_model_dir=\"/data\"\,org_id=$ORG_ID\)  (eb427ee8) took 1.439µs
1.5643062351040921e+09  info    * Pushing with cache ID eb427ee8
1.5643062351046875e+09  info    * Moving directories [] to /makisu-storage/sandbox/sandbox590207551/stages/MA==
1.5643068379290779e+09  error   Failed to push cache: timeout waiting for push
1.564306837930396e+09   info    Computed total image size 1189314536    {"total_image_size": 1189314536}
1.5643068379304373e+09  info    Successfully built image my-project/resolve-makisu-makisu:f597821dfabe48685ce3e1c0cd27071ebe833cd0
1.5643068379305375e+09  info    * Started pushing image gcr.io/my-project/resolve-makisu-makisu:f597821dfabe48685ce3e1c0cd27071ebe833cd0
1.564306838519872e+09   info    * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:bc9ab73e5b14b9fbd3687a4d8c1f1360533d6ee9ffc3f5ecc6630794b40257b7
1.5643068385219464e+09  info    * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:e5c3f8c317dc30af45021092a3d76f16ba7aa1ee5f18fec742c84d4960818580
1.564306838522682e+09   info    * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:193a6306c92af328dbd41bbbd3200a2c90802624cccfe5725223324428110d7f
1.5643068387359588e+09  info    * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:6598fc9d11d10365ac9281071a87930a2382ee31d026f1b6d432717b31db387c
1.5643068387402036e+09  info    * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:72744d0a318b0788001cc4f5f83c6847ba4b753307fadd046b508bbc41eb9e29
1.564306838744728e+09   info    * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:a587a86c9dcb9df6584180042becf21e36ecd8b460a761711227b4b06889a005
1.564306838940623e+09   info    * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:4b1d9004d467b4e710d770a881df027df7e5e7e4629f6e473760893ffc1a667f
1.5643068389598777e+09  info    * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:93612f47cdc374d0b33057b9e71eac173ac469da3e1a631dc8a32ba6986a408a
1.5643068389639986e+09  info    * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:1bc4b4b508703799ef67a807dacce4736045e642e87bcd49871cd0f23e7f5b8b
1.5643068393929317e+09  info    * Started pushing layer sha256:d3fc069215c7ba6059cf21b0b463c00c996710b66fe4fee5339b20b83b6778d0
1.5643068393939884e+09  info    * Started pushing layer sha256:34d73d444672f815723843b77fbb692893dc1e67fa779eb692b029994157ae74
1.5643068394151993e+09  info    * Started pushing layer sha256:6f7445cbad659141bfb874ad19a9704bd25b43417e469e7ba697fda138bc8009
1.5643068411155717e+09  info    * Finished pushing layer sha256:34d73d444672f815723843b77fbb692893dc1e67fa779eb692b029994157ae74
1.5643068413501308e+09  info    * Finished pushing layer sha256:6f7445cbad659141bfb874ad19a9704bd25b43417e469e7ba697fda138bc8009
1.564306841523365e+09   info    * Finished pushing layer sha256:d3fc069215c7ba6059cf21b0b463c00c996710b66fe4fee5339b20b83b6778d0
1.564306841618725e+09   info    * Started pushing layer sha256:8a041c71e7ccc1418342afce952b8115ea1351db58f128c06b7beece6ebe561f
1.5643068418106127e+09  info    * Started pushing layer sha256:b8a07cfb875b90ae3fd5d667dd651b0feba0d5956836b30e9ed9fb262fd68832
1.564306841900909e+09   info    * Started pushing layer sha256:e7d8cecf15ca1d353859cb01d3dc981d83c2688f67c0c28e63a4a5059b6e7d7b
1.5643068433530157e+09  info    * Finished pushing layer sha256:8a041c71e7ccc1418342afce952b8115ea1351db58f128c06b7beece6ebe561f
1.5643068436314905e+09  info    * Finished pushing layer sha256:b8a07cfb875b90ae3fd5d667dd651b0feba0d5956836b30e9ed9fb262fd68832
1.564306843823328e+09   info    * Started pushing layer sha256:5a48e99eaaff29fe6775cd72fb0457da90977cbdb3a66acaafa5a9dcd4723d2a
1.5643068440943959e+09  info    * Started pushing image config sha256:c9e8fec1c953d0b0e8b763953761d7a62ceb486c04099b42615e340769e0525b
1.5643068457586443e+09  info    * Finished pushing layer sha256:5a48e99eaaff29fe6775cd72fb0457da90977cbdb3a66acaafa5a9dcd4723d2a
1.5643068459129927e+09  info    * Finished pushing image config sha256:c9e8fec1c953d0b0e8b763953761d7a62ceb486c04099b42615e340769e0525b
1.564306875721196e+09   info    * Finished pushing layer sha256:e7d8cecf15ca1d353859cb01d3dc981d83c2688f67c0c28e63a4a5059b6e7d7b
1.564306876771674e+09   info    * Finished pushing image gcr.io/my-project/resolve-makisu-makisu:f597821dfabe48685ce3e1c0cd27071ebe833cd0 in 38.841108748s
1.564306876772124e+09   info    Successfully pushed gcr.io/my-project/resolve-makisu-makisu:f597821dfabe48685ce3e1c0cd27071ebe833cd0 to gcr.io
1.5643068767724044e+09  info    Finished building my-project/resolve-makisu-makisu:f597821dfabe48685ce3e1c0cd27071ebe833cd0

As for the rate limits: I think that is highly unlikely. I encountered this issue mostly when experimenting with makisu off-hours where we don't have much activity anyways, but even during the day I doubt we'd even get close to the rate limits.

Could it have to do somethign with the http cache though?