search

uber-archive / makisu

Fast and flexible Docker image building tool, works in unprivileged containerized environments like Mesos and Kubernetes.
Apache License 2.0
2.41k stars 154 forks source link

GCR image push failure #224

Closed Angry-Potato closed 5 years ago

Angry-Potato commented 5 years ago

I think this relates to #102.

Context :thinking:

I am running a Kubernetes (1.12.7-gke.10) cluster on GKE.

I use Helm (v2.12.1) to deploy Jenkins chart (1.1.16) and Redis-HA chart (3.4.1) from https://kubernetes-charts.storage.googleapis.com to this cluster.

I want to use Makisu to build images in Jenkins jobs, and push them to GCR.

To attempt to identify whether this is a regression or not, I have run jenkins jobs first using makisu's latest master version, and then using makisu 0.1.10, to compare.

commonalities

I was attempting to build the same Dockerfile for all builds: Dockerfile

The build always takes ~15mins, I expected it to be faster, unsure if this is normal.

master

I built latest makisu master in this image: Dockerfile (on dockerhub here)

And used this Jenkinsfile: Jenkinsfile.makisu-master_92443c6

Which yielded this output: makisu_master_92443c6_image_push_fail.log (note the usage of the --preserve-root flag)

Final makisu error in the output log is (I formatted this error a bit, non-whitespace is unchanged):

1.5579089264654338e+09 error
  failed to push image:
    failed to push image:
      push layer sha256:cb80490fc7e8104fbbfb21e07ba24065f170c487694da0d24621a48cfb26fe40:
        push layer content sha256:cb80490fc7e8104fbbfb21e07ba24065f170c487694da0d24621a48cfb26fe40:
          push layer chunk:
            send push chunk request:
              PATCH https://eu.gcr.io/v2/my-gcp-project-id/my-app/blobs/uploads/ABTmro4Gi53LPhetDZ-tacKHzYLnLdxA6oMndAE034xlsWGxe-VlKxpTYi_76Q_fi60KdoNVxkIs2ya6XhW-dWE8Lj6bR3-iE94aES-Ax0F3QUFhVuKseCri_my-T3Tp0X5-hzwuW4IhxhptzO_6OVTPsJunAG-AZjwddFpW-wTZEaR_NSkxbsHAVigl 404:
                {"errors":[{"code":"BLOB_UPLOAD_UNKNOWN","message":"Stream upload had partial digest data."}]}

0.1.10

I used this Jenkinsfile: Jenkinsfile.makisu-alpine:v0.1.10

Which yielded this output: makisu_v0.1.10_image_push_fail.log

The final makisu error in the output log is on line 37174, after that it's just the jenkins probe fail spam. The final makisu error is (again, formatted a bit but essentially unchanged):

1.55791263796012e+09    error
  failed to push image:
    failed to push image:
      push layer sha256:90ed2e38c749a7766b7c8eab92175bba09ebd564657abdac5d2db553ba32afc2:
        push layer content sha256:90ed2e38c749a7766b7c8eab92175bba09ebd564657abdac5d2db553ba32afc2:
          push layer chunk:
            send push chunk request:
              PATCH https://eu.gcr.io/v2/my-gcp-project-id/my-app/blobs/uploads/ABTmro4a5ZOcB4QonhP13tQvYMXAj6c81J7bBojBRM8bQFehRHV5Z5Q1EQaMnk2bsghsyXiRNEBWu-vQfd8qcbc6BQ0K380qWZuk8qWyWl7JCdUCJDBStRDLjKrhtZfJTzS5aW5GwBSOFVUR3rxQbOX7aa4zcOHSZZ9WmC6sFVU9eHsNZmK81SUAUVSH 404:
                {"errors":[{"code":"BLOB_UPLOAD_UNKNOWN","message":"Stream upload had partial digest data."}]}
Rowern commented 5 years ago

Here are the "cleaned" logs (removed any sh: .* not found errors for lisibility.

Master logs (cleaned)

makisu_master_92443c6_image_push_fail_cleaned.log

v1.10 (cleaned)

makisu_v0.1.10_image_push_fail_cleaned.log

Rowern commented 5 years ago

I'm able to reproduce (consistently) with a "big" image (361MB), using this dockerfile:

FROM mongo:3.4
Rowern commented 5 years ago

Some more debug infos, pushing the image from the above Dockerfile fails on GCR and ECR but does work on DockerHub. Tested with push_chunk: -1 and without. On AWS ECR The error is a little bit more explicit:

{"level":"error","ts":1557928361.36379,"msg":"failed to push image: failed to push image: push layer sha256:b82ef0d6a83f6ea013162bdf74676c5080655b811eb7718c5f6aadea96bf4b32: push layer content sha256:b82ef0d6a83f6ea013162bdf74676c5080655b811eb7718c5f6aadea96bf4b32: push layer chunk: send push chunk request: PATCH https://account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/ecr-name/blobs/uploads/4eb55de2-0bb9-9855-3037-f2cd1e41f89c 400: {\"errors\":[{\"code\":\"BLOB_UPLOAD_INVALID\",\"message\":\"First byte of the layer part is 0 instead of 52428800\"}]}\n"}
Rowern commented 5 years ago

While trying to debug this issue I found: https://github.com/GoogleContainerTools/kaniko/issues/607 and https://github.com/google/go-containerregistry/issues/431. I tried to add some retry here: https://github.com/uber/makisu/blob/92443c6b46d0fed7473eb2f71a739d265ecf9016/lib/registry/client.go#L173-L176 like this:

            retries := 4
            var err error
            for {
                retries--
                if err = c.PushLayer(l); err != nil {
                    log.Warnf("* Pushing layer %s failed (error: %s), retrying", l, err)
                }
                if retries < 0 {
                    break
                }
            }
            if err != nil {
                multiError.Add(fmt.Errorf("push layer %s: %s", l, err))
                workers.Stop()
                return
            }

But it does not work and keep telling me: {\"errors\":[{\"code\":\"BLOB_UPLOAD_INVALID\",\"message\":\"First byte of the layer part is 0 instead of 52428800\"}]}. The weird thing is that it tells us that when doing the request for the chunk starting at 52428800 (Pushing layer to URL: https://account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/project-name/blobs/uploads/46b55e24-c311-793f-47f0-9698e34d40b2, start: 52428800, end: 104857598)

I might be able to continue debugging tomorrow but I'm running thin on how to pinpoint the source of this error.

yiranwang52 commented 5 years ago

Maybe because Makisu was trying to push an empty layer but it shouldn't?

Rowern commented 5 years ago

I'm working on this branch here: https://github.com/Rowern/makisu/tree/debug-registry-blob-invalid It produces this output when pushing the mongo:3.4 docker image:

API server listening at: 127.0.0.1:44845
{"level":"debug","ts":1557993564.660289,"msg":"Cannot init mountmanager, /proc/mounts does not exist"}
{"level":"info","ts":1557993564.66066,"msg":"Starting Makisu build (version=master-unreleased)"}
{"level":"warn","ts":1557993564.7238212,"msg":"Failed to load cached manifest: file does not exist"}
{"level":"warn","ts":1557993564.72387,"msg":"Failed to cleanup cached manifest: file does not exist"}
{"level":"warn","ts":1557993564.732987,"msg":"Failed to load cached manifest: file does not exist"}
{"level":"warn","ts":1557993564.733027,"msg":"Failed to cleanup cached manifest: file does not exist"}
{"level":"warn","ts":1557993564.734605,"msg":"Failed to load cached manifest: file does not exist"}
{"level":"warn","ts":1557993564.734646,"msg":"Failed to cleanup cached manifest: file does not exist"}
{"level":"warn","ts":1557993564.7362418,"msg":"Failed to load cached manifest: file does not exist"}
{"level":"warn","ts":1557993564.7362711,"msg":"Failed to cleanup cached manifest: file does not exist"}
{"level":"info","ts":1557993564.752471,"msg":"Using build context: /Users/cometresarrieu/go/src/github.com/uber/makisu"}
{"level":"info","ts":1557993564.753301,"msg":"Using local file at /tmp/makisu-storage/cache_key_value.json for cacheID storage"}
{"level":"info","ts":1557993564.753877,"msg":"* Stage 1/1 : (alias=0,latestfetched=-1)"}
{"level":"info","ts":1557993564.753912,"msg":"* Step 1/1 (commit) : FROM mongo:3.4  (93740dd9)"}
{"level":"info","ts":1557993564.754046,"msg":"* Started pulling image index.docker.io/library/mongo:3.4"}
{"level":"info","ts":1557993567.385581,"msg":"* Started pulling layer index.docker.io/library/mongo:sha256:59856638ac9f32d4caa0f5761b2597fe251642786fdfe1b917ddbb074b890c29"}
{"level":"info","ts":1557993567.839465,"msg":"* Started pulling layer index.docker.io/library/mongo:sha256:6f317d6d954b9a59c54b2cb09e1f30cd3e872796e431cd2ceac5ed570beb2939"}
{"level":"info","ts":1557993567.8416378,"msg":"* Started pulling layer index.docker.io/library/mongo:sha256:9ff7e2e5f967fb9c4e8099e63508ab0dddebe3f820d08ca7fd568431b0d10c0e"}
{"level":"info","ts":1557993568.177674,"msg":"* Finished pulling layer library/mongo:59856638ac9f32d4caa0f5761b2597fe251642786fdfe1b917ddbb074b890c29"}
{"level":"info","ts":1557993568.193596,"msg":"* Finished pulling layer library/mongo:6f317d6d954b9a59c54b2cb09e1f30cd3e872796e431cd2ceac5ed570beb2939"}
{"level":"info","ts":1557993568.266167,"msg":"* Started pulling layer index.docker.io/library/mongo:sha256:a9dde5e2a643eca8fde0eed52f4aed31f3ecd9c1b2f24d5e3729cd8d2ae68177"}
{"level":"info","ts":1557993568.343224,"msg":"* Started pulling layer index.docker.io/library/mongo:sha256:815c6aedc001b0c406021809f6ddda02eafc2db0af37d82e719a469ff691a5bb"}
{"level":"info","ts":1557993568.529682,"msg":"* Finished pulling layer library/mongo:a9dde5e2a643eca8fde0eed52f4aed31f3ecd9c1b2f24d5e3729cd8d2ae68177"}
{"level":"info","ts":1557993568.5545752,"msg":"* Finished pulling layer library/mongo:815c6aedc001b0c406021809f6ddda02eafc2db0af37d82e719a469ff691a5bb"}
{"level":"info","ts":1557993568.614536,"msg":"* Started pulling layer index.docker.io/library/mongo:sha256:8566b2594855b4802bf2ec5a3cef61efe1d3ed97cfa6a747f397b7bb08db3c5b"}
{"level":"info","ts":1557993568.6395452,"msg":"* Started pulling layer index.docker.io/library/mongo:sha256:01c9fe451980007df2a3585e68580ed005adb0a8f2bb3e85af75afa8034e5d75"}
{"level":"info","ts":1557993570.45348,"msg":"* Finished pulling layer library/mongo:01c9fe451980007df2a3585e68580ed005adb0a8f2bb3e85af75afa8034e5d75"}
{"level":"info","ts":1557993570.5397592,"msg":"* Started pulling layer index.docker.io/library/mongo:sha256:5c9e7bc12ceae7b610677a07d228667d9ec99d24069d391d3bda4a29370ef083"}
{"level":"info","ts":1557993570.769061,"msg":"* Finished pulling layer library/mongo:5c9e7bc12ceae7b610677a07d228667d9ec99d24069d391d3bda4a29370ef083"}
{"level":"info","ts":1557993570.8534029,"msg":"* Started pulling layer index.docker.io/library/mongo:sha256:fcf515753313f6850ce93164f1160967ecbdb1fdb16d0dd3225475cf505c0806"}
{"level":"info","ts":1557993571.332363,"msg":"* Finished pulling layer library/mongo:fcf515753313f6850ce93164f1160967ecbdb1fdb16d0dd3225475cf505c0806"}
{"level":"info","ts":1557993571.4195142,"msg":"* Started pulling layer index.docker.io/library/mongo:sha256:106481e26b7980c2cc564379d7c701304efa5ad2227a6b89eebf3f3f5ca98037"}
{"level":"info","ts":1557993571.843054,"msg":"* Finished pulling layer library/mongo:8566b2594855b4802bf2ec5a3cef61efe1d3ed97cfa6a747f397b7bb08db3c5b"}
{"level":"info","ts":1557993571.910165,"msg":"* Finished pulling layer library/mongo:106481e26b7980c2cc564379d7c701304efa5ad2227a6b89eebf3f3f5ca98037"}
{"level":"info","ts":1557993571.969546,"msg":"* Started pulling layer index.docker.io/library/mongo:sha256:44d18c76895dddf7c8bc9b840b3491c47cbea285840d856b49ca32e580cbd88d"}
{"level":"info","ts":1557993572.176403,"msg":"* Started pulling layer index.docker.io/library/mongo:sha256:21d728162b5d8a69b01ca2cd38b070ea3700038a8ab1b3305ef8857df2404615"}
{"level":"info","ts":1557993572.3849049,"msg":"* Finished pulling layer library/mongo:21d728162b5d8a69b01ca2cd38b070ea3700038a8ab1b3305ef8857df2404615"}
{"level":"info","ts":1557993572.470146,"msg":"* Started pulling layer index.docker.io/library/mongo:sha256:a75a38b109f93f3a75f77af3d498be38be2e1d9e5e2a06776522a4c606df74cc"}
{"level":"info","ts":1557993572.6943989,"msg":"* Finished pulling layer library/mongo:a75a38b109f93f3a75f77af3d498be38be2e1d9e5e2a06776522a4c606df74cc"}
{"level":"info","ts":1557993572.779118,"msg":"* Started pulling layer index.docker.io/library/mongo:sha256:d470115d6109817b3f0f11f840c7ca8d62da6f029772dd0cd4ba0349755c9508"}
{"level":"info","ts":1557993573.257602,"msg":"* Finished pulling layer library/mongo:d470115d6109817b3f0f11f840c7ca8d62da6f029772dd0cd4ba0349755c9508"}
{"level":"info","ts":1557993573.341886,"msg":"* Started pulling layer index.docker.io/library/mongo:sha256:3e07e22f0dbf3dcd953e42e838460ca76726c5c04a1020d87d82ff18a41ca488"}
{"level":"info","ts":1557993573.8441951,"msg":"* Finished pulling layer library/mongo:3e07e22f0dbf3dcd953e42e838460ca76726c5c04a1020d87d82ff18a41ca488"}
{"level":"info","ts":1557993605.0920691,"msg":"* Finished pulling layer library/mongo:9ff7e2e5f967fb9c4e8099e63508ab0dddebe3f820d08ca7fd568431b0d10c0e"}
{"level":"info","ts":1557993640.6176891,"msg":"* Finished pulling layer library/mongo:44d18c76895dddf7c8bc9b840b3491c47cbea285840d856b49ca32e580cbd88d"}
{"level":"info","ts":1557993640.6178482,"msg":"* Finished pulling image index.docker.io/library/mongo:3.4 in 1m15.863398799s"}
{"level":"info","ts":1557993640.621834,"msg":"* Processing FROM layer 9ff7e2e5f967fb9c4e8099e63508ab0dddebe3f820d08ca7fd568431b0d10c0e"}
{"level":"info","ts":1557993642.465863,"msg":"* Merged 5447 headers from tar to memfs"}
{"level":"info","ts":1557993642.468092,"msg":"* Processing FROM layer 59856638ac9f32d4caa0f5761b2597fe251642786fdfe1b917ddbb074b890c29"}
{"level":"info","ts":1557993642.469355,"msg":"* Merged 20 headers from tar to memfs"}
{"level":"info","ts":1557993642.474043,"msg":"* Processing FROM layer 6f317d6d954b9a59c54b2cb09e1f30cd3e872796e431cd2ceac5ed570beb2939"}
{"level":"info","ts":1557993642.475273,"msg":"* Merged 21 headers from tar to memfs"}
{"level":"info","ts":1557993642.4775522,"msg":"* Processing FROM layer a9dde5e2a643eca8fde0eed52f4aed31f3ecd9c1b2f24d5e3729cd8d2ae68177"}
{"level":"info","ts":1557993642.478036,"msg":"* Merged 3 headers from tar to memfs"}
{"level":"info","ts":1557993642.479311,"msg":"* Processing FROM layer 815c6aedc001b0c406021809f6ddda02eafc2db0af37d82e719a469ff691a5bb"}
{"level":"info","ts":1557993642.480355,"msg":"* Merged 13 headers from tar to memfs"}
{"level":"info","ts":1557993642.480972,"msg":"* Processing FROM layer 8566b2594855b4802bf2ec5a3cef61efe1d3ed97cfa6a747f397b7bb08db3c5b"}
{"level":"info","ts":1557993642.612628,"msg":"* Merged 851 headers from tar to memfs"}
{"level":"info","ts":1557993642.613437,"msg":"* Processing FROM layer 01c9fe451980007df2a3585e68580ed005adb0a8f2bb3e85af75afa8034e5d75"}
{"level":"info","ts":1557993642.6672199,"msg":"* Merged 42 headers from tar to memfs"}
{"level":"info","ts":1557993642.667946,"msg":"* Processing FROM layer 5c9e7bc12ceae7b610677a07d228667d9ec99d24069d391d3bda4a29370ef083"}
{"level":"info","ts":1557993642.66816,"msg":"* Merged 1 headers from tar to memfs"}
{"level":"info","ts":1557993642.6693928,"msg":"* Processing FROM layer fcf515753313f6850ce93164f1160967ecbdb1fdb16d0dd3225475cf505c0806"}
{"level":"info","ts":1557993642.669912,"msg":"* Merged 5 headers from tar to memfs"}
{"level":"info","ts":1557993642.67398,"msg":"* Processing FROM layer 106481e26b7980c2cc564379d7c701304efa5ad2227a6b89eebf3f3f5ca98037"}
{"level":"info","ts":1557993642.674452,"msg":"* Merged 4 headers from tar to memfs"}
{"level":"info","ts":1557993642.6752808,"msg":"* Processing FROM layer 44d18c76895dddf7c8bc9b840b3491c47cbea285840d856b49ca32e580cbd88d"}
{"level":"info","ts":1557993646.9588811,"msg":"* Merged 104 headers from tar to memfs"}
{"level":"info","ts":1557993646.959532,"msg":"* Processing FROM layer 21d728162b5d8a69b01ca2cd38b070ea3700038a8ab1b3305ef8857df2404615"}
{"level":"info","ts":1557993646.9598432,"msg":"* Merged 3 headers from tar to memfs"}
{"level":"info","ts":1557993646.964064,"msg":"* Processing FROM layer a75a38b109f93f3a75f77af3d498be38be2e1d9e5e2a06776522a4c606df74cc"}
{"level":"info","ts":1557993646.964582,"msg":"* Merged 4 headers from tar to memfs"}
{"level":"info","ts":1557993646.9653518,"msg":"* Processing FROM layer d470115d6109817b3f0f11f840c7ca8d62da6f029772dd0cd4ba0349755c9508"}
{"level":"info","ts":1557993646.96556,"msg":"* Merged 1 headers from tar to memfs"}
{"level":"info","ts":1557993646.96559,"msg":"* Execute FROM mongo:3.4  (93740dd9) took 1m22.211249658s"}
{"level":"info","ts":1557993646.96813,"msg":"Computed total image size 167485998","total_image_size":167485998}
{"level":"info","ts":1557993646.968159,"msg":"Successfully built image my-project-name:1234"}
{"level":"info","ts":1557993646.9682932,"msg":"* Started pushing image my-account-id.dkr.ecr.eu-west-3.amazonaws.com/my-project-name:1234"}
{"level":"info","ts":1557993648.005561,"msg":"* Image my-account-id.dkr.ecr.eu-west-3.amazonaws.com/my-project-name:1234 already exists, overwriting"}
{"level":"info","ts":1557993649.380865,"msg":"* Started pushing layer sha256:6f317d6d954b9a59c54b2cb09e1f30cd3e872796e431cd2ceac5ed570beb2939"}
{"level":"debug","ts":1557993649.3809762,"msg":"  To URL https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/e0b55fd4-3a6c-a170-fb28-aaf5f13b35f7"}
{"level":"info","ts":1557993649.384564,"msg":"* Started pushing layer sha256:59856638ac9f32d4caa0f5761b2597fe251642786fdfe1b917ddbb074b890c29"}
{"level":"debug","ts":1557993649.384691,"msg":"  To URL https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/42b55fd4-3a6d-55be-f935-2bca0832b6d7"}
{"level":"info","ts":1557993649.42264,"msg":"* Started pushing layer sha256:9ff7e2e5f967fb9c4e8099e63508ab0dddebe3f820d08ca7fd568431b0d10c0e"}
{"level":"debug","ts":1557993649.422737,"msg":"  To URL https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/30b55fd4-3a81-f726-21e2-38cf1215bd95"}
{"level":"debug","ts":1557993649.987666,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/e0b55fd4-3a6c-a170-fb28-aaf5f13b35f7, start: 0, end: 525"}
{"level":"debug","ts":1557993649.989685,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/42b55fd4-3a6d-55be-f935-2bca0832b6d7, start: 0, end: 847"}
{"level":"debug","ts":1557993650.042457,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/30b55fd4-3a81-f726-21e2-38cf1215bd95, start: 0, end: 43767986"}
{"level":"debug","ts":1557993650.1044621,"msg":"  Finished pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/42b55fd4-3a6d-55be-f935-2bca0832b6d7, start: 0, end: 847"}
{"level":"debug","ts":1557993650.1155589,"msg":"  Finished pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/e0b55fd4-3a6c-a170-fb28-aaf5f13b35f7, start: 0, end: 525"}
{"level":"info","ts":1557993651.0176058,"msg":"* Finished pushing layer sha256:59856638ac9f32d4caa0f5761b2597fe251642786fdfe1b917ddbb074b890c29"}
{"level":"info","ts":1557993651.772971,"msg":"* Finished pushing layer sha256:6f317d6d954b9a59c54b2cb09e1f30cd3e872796e431cd2ceac5ed570beb2939"}
{"level":"info","ts":1557993653.640179,"msg":"* Started pushing layer sha256:815c6aedc001b0c406021809f6ddda02eafc2db0af37d82e719a469ff691a5bb"}
{"level":"debug","ts":1557993653.64022,"msg":"  To URL https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/46b55fd4-42bf-0c70-7a23-16253db9fa6b"}
{"level":"info","ts":1557993654.062161,"msg":"* Started pushing layer sha256:a9dde5e2a643eca8fde0eed52f4aed31f3ecd9c1b2f24d5e3729cd8d2ae68177"}
{"level":"debug","ts":1557993654.0622258,"msg":"  To URL https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/b6b55fd4-4392-b934-2d55-e13d4a8a588b"}
{"level":"debug","ts":1557993654.2297652,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/46b55fd4-42bf-0c70-7a23-16253db9fa6b, start: 0, end: 1983"}
{"level":"debug","ts":1557993654.349422,"msg":"  Finished pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/46b55fd4-42bf-0c70-7a23-16253db9fa6b, start: 0, end: 1983"}
{"level":"debug","ts":1557993654.65244,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/b6b55fd4-4392-b934-2d55-e13d4a8a588b, start: 0, end: 167"}
{"level":"debug","ts":1557993654.780993,"msg":"  Finished pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/b6b55fd4-4392-b934-2d55-e13d4a8a588b, start: 0, end: 167"}
{"level":"info","ts":1557993655.0776598,"msg":"* Finished pushing layer sha256:815c6aedc001b0c406021809f6ddda02eafc2db0af37d82e719a469ff691a5bb"}
{"level":"info","ts":1557993655.560334,"msg":"* Finished pushing layer sha256:a9dde5e2a643eca8fde0eed52f4aed31f3ecd9c1b2f24d5e3729cd8d2ae68177"}
{"level":"debug","ts":1557993656.264617,"msg":"  Finished pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/30b55fd4-3a81-f726-21e2-38cf1215bd95, start: 0, end: 43767986"}
{"level":"info","ts":1557993656.367234,"msg":"* Started pushing layer sha256:8566b2594855b4802bf2ec5a3cef61efe1d3ed97cfa6a747f397b7bb08db3c5b"}
{"level":"debug","ts":1557993656.367317,"msg":"  To URL https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2
/my-project-name/blobs/uploads/48b55fd4-4811-c727-8f34-18bac4484324"}
{"level":"info","ts":1557993656.895201,"msg":"* Started pushing layer sha256:01c9fe451980007df2a3585e68580ed005adb0a8f2bb3e85af75afa8034e5d75"}
{"level":"debug","ts":1557993656.895246,"msg":"  To URL https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/14b55fd4-491a-a65f-bfde-1d5c01bec440"}
{"level":"debug","ts":1557993656.9413881,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/48b55fd4-4811-c727-8f34-18bac4484324, start: 0, end: 2945615"}
{"level":"info","ts":1557993657.043583,"msg":"* Finished pushing layer sha256:9ff7e2e5f967fb9c4e8099e63508ab0dddebe3f820d08ca7fd568431b0d10c0e"}
{"level":"debug","ts":1557993657.4761732,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/14b55fd4-491a-a65f-bfde-1d5c01bec440, start: 0, end: 1151232"}
{"level":"debug","ts":1557993657.569986,"msg":"  Finished pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/48b55fd4-4811-c727-8f34-18bac4484324, start: 0, end: 2945615"}
{"level":"debug","ts":1557993657.792447,"msg":"  Finished pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/14b55fd4-491a-a65f-bfde-1d5c01bec440, start: 0, end: 1151232"}
{"level":"info","ts":1557993658.3018692,"msg":"* Finished pushing layer sha256:8566b2594855b4802bf2ec5a3cef61efe1d3ed97cfa6a747f397b7bb08db3c5b"}
{"level":"info","ts":1557993658.470986,"msg":"* Started pushing layer sha256:5c9e7bc12ceae7b610677a07d228667d9ec99d24069d391d3bda4a29370ef083"}
{"level":"debug","ts":1557993658.4710479,"msg":"  To URL https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/94b55fd4-4c2c-8368-cbf5-3fea1417903f"}
{"level":"info","ts":1557993658.55175,"msg":"* Finished pushing layer sha256:01c9fe451980007df2a3585e68580ed005adb0a8f2bb3e85af75afa8034e5d75"}
{"level":"debug","ts":1557993659.045116,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/94b55fd4-4c2c-8368-cbf5-3fea1417903f, start: 0, end: 114"}
{"level":"debug","ts":1557993659.152452,"msg":"  Finished pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/94b55fd4-4c2c-8368-cbf5-3fea1417903f, start: 0, end: 114"}
{"level":"info","ts":1557993659.6002932,"msg":"* Started pushing layer sha256:fcf515753313f6850ce93164f1160967ecbdb1fdb16d0dd3225475cf505c0806"}
{"level":"debug","ts":1557993659.6004128,"msg":"  To URL https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/80b55fd4-4e63-c64b-94cf-69948072a93f"}
{"level":"info","ts":1557993659.8805592,"msg":"* Finished pushing layer sha256:5c9e7bc12ceae7b610677a07d228667d9ec99d24069d391d3bda4a29370ef083"}
{"level":"info","ts":1557993659.887441,"msg":"* Started pushing layer sha256:106481e26b7980c2cc564379d7c701304efa5ad2227a6b89eebf3f3f5ca98037"}
{"level":"debug","ts":1557993659.887498,"msg":"  To URL https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/94b55fd4-4ef0-8e7c-4f8f-80420cdd3560"}
{"level":"debug","ts":1557993660.1855621,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/80b55fd4-4e63-c64b-94cf-69948072a93f, start: 0, end: 1997"}
{"level":"debug","ts":1557993660.297547,"msg":"  Finished pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/80b55fd4-4e63-c64b-94cf-69948072a93f, start: 0, end: 1997"}
{"level":"debug","ts":1557993660.496373,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/94b55fd4-4ef0-8e7c-4f8f-80420cdd3560, start: 0, end: 237"}
{"level":"debug","ts":1557993660.621716,"msg":"  Finished pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/94b55fd4-4ef0-8e7c-4f8f-80420cdd3560, start: 0, end: 237"}
{"level":"info","ts":1557993661.000391,"msg":"* Finished pushing layer sha256:fcf515753313f6850ce93164f1160967ecbdb1fdb16d0dd3225475cf505c0806"}
{"level":"info","ts":1557993661.201855,"msg":"* Started pushing layer sha256:44d18c76895dddf7c8bc9b840b3491c47cbea285840d856b49ca32e580cbd88d"}
{"level":"debug","ts":1557993661.20189,"msg":"  To URL https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/0eb55fd4-5183-d09f-4515-08ca22954536"}
{"level":"info","ts":1557993661.36271,"msg":"* Finished pushing layer sha256:106481e26b7980c2cc564379d7c701304efa5ad2227a6b89eebf3f3f5ca98037"}
{"level":"debug","ts":1557993661.7864408,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/0eb55fd4-5183-d09f-4515-08ca22954536, start: 0, end: 52428799"}
{"level":"info","ts":1557993664.899122,"msg":"* Started pushing layer sha256:21d728162b5d8a69b01ca2cd38b070ea3700038a8ab1b3305ef8857df2404615"}
{"level":"debug","ts":1557993664.8991811,"msg":"  To URL https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/a2b55fd4-58bc-ee56-9c9c-a869cbe06802"}
{"level":"info","ts":1557993665.4465501,"msg":"* Started pushing layer sha256:a75a38b109f93f3a75f77af3d498be38be2e1d9e5e2a06776522a4c606df74cc"}
{"level":"debug","ts":1557993665.446611,"msg":"  To URL https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/62b55fd4-59ce-1470-45eb-e7c01d8368d0"}
{"level":"debug","ts":1557993665.467445,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/a2b55fd4-58bc-ee56-9c9c-a869cbe06802, start: 0, end: 140"}
{"level":"debug","ts":1557993665.587256,"msg":"  Finished pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/a2b55fd4-58bc-ee56-9c9c-a869cbe06802, start: 0, end: 140"}
{"level":"debug","ts":1557993666.036503,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/62b55fd4-59ce-1470-45eb-e7c01d8368d0, start: 0, end: 3916"}
{"level":"debug","ts":1557993666.1512718,"msg":"  Finished pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/62b55fd4-59ce-1470-45eb-e7c01d8368d0, start: 0, end: 3916"}
{"level":"info","ts":1557993666.352727,"msg":"* Finished pushing layer sha256:21d728162b5d8a69b01ca2cd38b070ea3700038a8ab1b3305ef8857df2404615"}
{"level":"info","ts":1557993666.894576,"msg":"* Finished pushing layer sha256:a75a38b109f93f3a75f77af3d498be38be2e1d9e5e2a06776522a4c606df74cc"}
{"level":"info","ts":1557993667.663231,"msg":"* Started pushing layer sha256:d470115d6109817b3f0f11f840c7ca8d62da6f029772dd0cd4ba0349755c9508"}
{"level":"debug","ts":1557993667.6633348,"msg":"  To URL https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/8ab55fd4-5e22-f0f6-af10-b47b92aa6b85"}
{"level":"info","ts":1557993668.198863,"msg":"* Started pushing image config sha256:9c3cdb33cdfaa79f0f5bb3da0ff7f80a6c084c477b9087935bc5974aec33a891"}
{"level":"debug","ts":1557993668.198955,"msg":"  To URL https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/f6b55fd4-5f2b-8eb5-b5bc-b08a410a76ad"}
{"level":"debug","ts":1557993668.2137668,"msg":"  Finished pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/0eb55fd4-5183-d09f-4515-08ca22954536, start: 0, end: 52428799"}
{"level":"debug","ts":1557993668.24223,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/8ab55fd4-5e22-f0f6-af10-b47b92aa6b85, start: 0, end: 120"}
{"level":"debug","ts":1557993668.336364,"msg":"  Finished pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/8ab55fd4-5e22-f0f6-af10-b47b92aa6b85, start: 0, end: 120"}
{"level":"debug","ts":1557993668.7785332,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/f6b55fd4-5f2b-8eb5-b5bc-b08a410a76ad, start: 0, end: 3597"}
{"level":"debug","ts":1557993668.804963,"msg":"  Pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/0eb55fd4-5183-d09f-4515-08ca22954536, start: 52428800, end: 104857598"}
{"level":"debug","ts":1557993668.8810802,"msg":"  Finished pushing layer to URL: https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/f6b55fd4-5f2b-8eb5-b5bc-b08a410a76ad, start: 0, end: 3597"}
{"level":"info","ts":1557993669.204235,"msg":"* Finished pushing layer sha256:d470115d6109817b3f0f11f840c7ca8d62da6f029772dd0cd4ba0349755c9508"}
{"level":"info","ts":1557993670.5050461,"msg":"* Finished pushing image config sha256:9c3cdb33cdfaa79f0f5bb3da0ff7f80a6c084c477b9087935bc5974aec33a891"}
{"level":"error","ts":1557993670.639991,"msg":"failed to push image: failed to push image: push layer sha256:44d18c76895dddf7c8bc9b840b3491c47cbea285840d856b49ca32e580cbd88d: push layer content sha256:44d18c76895dddf7c8bc9b840b3491c47cbea285840d856b49ca32e580cbd88d: push layer chunk: send push chunk request: PATCH https://my-account-id.dkr.ecr.eu-west-3.amazonaws.com/v2/my-project-name/blobs/uploads/0eb55fd4-5183-d09f-4515-08ca22954536 400: {\"errors\":[{\"code\":\"BLOB_UPLOAD_INVALID\",\"message\":\"First byte of the layer part is 0 instead of 52428800\"}]}\n"}

I don't know if that helps...

Rowern commented 5 years ago

As it works with docker we might be able to see what they do and mimic their behavior.

Here is my analysis of their upload flow (might be totally wrong, please correct me if its not):

What looks different than makisu is that they do "use" chunk but they most likely never "chunk" their uploads. I'm going to try with no chunk to see it fixes the issue.

Rowern commented 5 years ago

Also while playing with GCR and the headers used: instead of:

        "Host":           c.registry,
        "Content-Type":   "application/octet-stream",
        "Content-Length": fmt.Sprintf("%d", chunckSize),
        "Content-Range":  fmt.Sprintf("%d-%d", start, endIncluded),

I tried:

        "Host":           c.registry,
        "Content-Type":   "application/octet-stream",
        "Content-Length": fmt.Sprintf("%d", chunckSize),
        // "Content-Range":  fmt.Sprintf("%d-%d", start, endIncluded),
        "Range": fmt.Sprintf("bytes=%d-%d", start, endIncluded),

And got the error: {\"errors\":[{\"code\":\"UNSUPPORTED\",\"message\":\"Chunked upload is not supported.\"}]}

So maybe chunk upload as it is does not work for GCR because it is not supported by GCR (or maybe my headers where just putting GCR in a weird state).

Rowern commented 5 years ago

I found a fix (albeit a simple one) that works for my tests. The bug was caused by failing to Unmarshall the push_chunk: -1 when passing regirsty-config as JSON.

Rowern commented 5 years ago

Fix confirmed on slack by @Angry-Potato