Kaniko and proxy set up #2252

Open milica-nikolic opened 2 years ago

Hi everyone,

Server where runners for my pipeline which runs on GitLab are rigistered is behined proxy. I followed the documentation so that I can set up the build of image with Kaniko behind the proxy, but I still get an error which based on documentation is for missing configuration for proxy. Can someone tell me if there is a step that I may have left out or if there is something else that needs to be adjusted and I don't know about it. After build of image is successfully done I want to be able to push image to Harbor registry with latest tag.

Thank you for your help !

Below I will leave how the settings for that stage look like in pipeline:


10:execution (dev branch):
  stage: Build and Push Image
  variables:
    http_proxy: ${PROXY_ADDRESS}
    https_proxy: ${PROXY_ADDRESS}
    no_proxy: "127.0.0.1,localhost"
  needs:
    - job: 9:prep for build (dev branch)
      artifacts: true
  image:
    name: gcr.io/kaniko-project/executor:v1.9.0-debug
    entrypoint: [""]
  script:
    - LATEST_TAG=$(cat tags.txt)
    - echo "{\"auths\":{\"$HARBOR_REGISTRY\":{\"username\":\"$HARBOR_USER\",\"password\":\"$HARBOR_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --build-arg http_proxy=$http_proxy
      --build-arg https_proxy=$https_proxy
      --build-arg no_proxy=$no_proxy
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:$LATEST_TAG"
  only:
    refs:
      - dev
  tags:
    - dev
    - docker

The error I am getting:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "some address": creating push check transport for "some address" failed: Get "some address" Forbidden

I have exactly the same problem with a GitlabCI runner that need to go through a proxy to push the image.

Error error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "gitlab.domain.com:5050/path/to/image:0.0.1": creating push check transport for gitlab.domain.com:5050 failed: Get "https://gitlab.domain.com:5050/v2/": Forbidden

.gitlab-ci.yml

build:
  stage: build
  variables:
    http_proxy: http://PROXY:PORT
    https_proxy: http://PROXY:PORT
    HTTP_PROXY: http://PROXY:PORT
    HTTPS_PROXY: http://PROXY:PORT
  image:
    name: gcr.io/kaniko-project/executor:v1.14.0-debug
    entrypoint: [""]
  before_script:
    - echo -n "{\"proxies\":{\"default\":{\"httpProxy\":\"http://PROXY:PORT\",\"httpsProxy\":\"http://PROXY:PORT\",\"noProxy\":\"localhost,127.0.0.1\"}},\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | tr -d "\n" | base64)\"}}}"
        > /kaniko/.docker/config.json
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --build-arg "http_proxy=${http_proxy}"
      --build-arg "https_proxy=${https_proxy}"
      --dockerfile "${CI_PROJECT_DIR}"
      --destination "${CI_REGISTRY_IMAGE}:0.0.1"

I have tried a lot of things, verified that the env variables are well accessible. I also checked if when I do a --no-push option, the build and the download through the proxy inside the build is working fine (it is the case). The file /kaniko/.docker/config.json is well formatted when written inside the container.

It seems that only the checking push permission is unhappy with the proxy or do not use it at all.

I will be happy to do more tests if needed to actively help to resolve this issue.

@Xysto can you try using the --skip-push-permission-check flag (docs here - https://github.com/GoogleContainerTools/kaniko?tab=readme-ov-file#flag---skip-push-permission-check) and see if that is able to resolve this issue using a proxy? Currently it's not clear to me if Kaniko cannot auth properly with the proxy or only the authentication check (which the flag skips) has an issue

@aaron-prindle still have a forbidden at the end when it's trying to push.

error pushing image: failed to push to destination gitlab.domain.com:5050/path/to/image:0.0.1: Get "https://gitlab.domain.com:5050/v2/": Forbidden

Can it be due to this note in your readme.me ? Note: Please use v1 endpoint. See #1209 for more details But again I've tried all the workaround in #1209 to use v1 endpoint but the error was still using v2. Get "https://gitlab.domain.com:5050/v2/": Forbidden

I dived deeper and after some discussion with the proxy team, it seems that the 403: Forbidden was returned BY the proxy itself and not by the Gitlab registry. The TLS connections were not allowed on another port than 443. I was misled by the error message that directly pointed out the Gitlab registry. Kaniko uses well the http(s) env variables and everything works perfectly. This tool is awesome and answers to a lot of security concerns that we had. Sorry for the waste of time...

GoogleContainerTools / kaniko

Kaniko and proxy set up #2252