Pushing with kaniko is slow

[suleimi]

Actual behavior Pushing large images (e.g 5Gb) with kaniko to some https registries (e.g ecr, quay or gitlab contianer registry) has been identified to be quite slow when using kaniko . After some preliminary investigation (in https://gitlab.com/gitlab-org/gitlab/-/issues/241996 )it appears the slowness comes from the issue mentioned in https://github.com/golang/go/issues/47840 and is related to the http/2 implementation of net/http

Because of the slowness caused by https://github.com/golang/go/issues/47840 docker and buildx registry clients have opted to only push on http/1.1 connection. see: https://github.com/moby/buildkit/pull/1420

A few issues have been raised on (go-containerregistry) which kaniko uses that alludes to a similar http2 bottleneck:

• https://github.com/google/go-containerregistry/issues/1193
• https://github.com/google/go-containerregistry/issues/1210

Expected behavior

Pushing a 5Gb layer using kaniko to gitlab contianer registry, ecr or quay should not take > 30mins on arguably fast internet connections

To Reproduce Steps to reproduce the behavior:

1. Push a random 5Gb image to gitlab container registry
2. Push a random 5Gb image to gitlab container registry with http 2 disabled (by setting GODEBUG="http2client=0")
3. Observe it takes almost > 40mins with http 2 enabled

Additional Information

• Setup: https://gitlab.com/suleimiahmed/registry-speed-test/-/tree/test-ff?ref_type=heads

  • includes docker file
  • Any kaniko that defaults to http 2 has this issue

  Triage Notes for the Maintainers

  Description	Yes/No
  Please check if this a new feature you are proposing	- [ ]
  Please check if the build works in docker but not in kaniko	- [x]
  Please check if this error is seen when you use --cache flag	- [ ]
  Please check if your dockerfile is a multistage dockerfile	- [ ]

[slamer59]

From 40 minutes to 2 minutes with workaround... Thanks !

[stocktondr]

From 40 minutes to 2 minutes with workaround... Thanks !

We are having the same problem. The workaround also worked for us.

[zijiwork]

How to solve the problem of slow push in tekton kaniko

[keepdying]

We also saw improvements on builds while pushing caches and image to gcr and gar. From 13 mins to 7 mins. Thanks for workaround!

[psarka]

In my case it went from 38min to 2min. @zijiwork this is how you do it in gitlab:

build-ci-image:
    stage: build-ci-image
    variables:
         GODEBUG: "http2client=0"
    image:
        name: gcr.io/kaniko-project/executor:v1.9.0-debug
        entrypoint: [ "" ]
    script:
        - /kaniko/executor
            --context ...
            --dockerfile ...
            --destination ...

[aapjeisbaas]

Uploads to gitlab registry went from max 5 mbit/s to full line speed

build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  variables:
    GODEBUG: "http2client=0"
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"

[derekyle]

Is anyone aware of whether this bug persists in later version releases? I am running v1.23.0 and implemented this fix a few months ago. At the time, it made a very big difference, bringing our build process from around 30 minutes to less than 10. However, it seems I have started seeing longer and longer registry upload times. We have been using gitlab all along. Occasionally, it is taking 30 minutes to upload a 250mb image to our gitlab container registry.

[derekyle]

To add details and partially answer my own question, using Kaniko v1.23.2-debug, removing the GODEBUG: "http2client=0" workaround dropped my upload time for a 500mb image from about 15 minutes to 5 minutes. So it seems that maybe part of this issue was fixed. 5 minutes for 500mb still seems excessively slow to me, but it could be a gitlab container repository issue at this point.