"GET https://example.com/users/sign_in: unexpected status code 200 OK" when pushing to self-hosted GitLab with Kaniko #3146

Open andrew2758 opened 6 months ago

Actual behavior

When attempting to publish a Kaniko-built image to a self-hosted GitLab server (which I'll call example.com in this issue), we get the following error:

$ /kaniko/executor --context "${CI_PROJECT_DIR}" --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "registry-example.com:443/me/registry-test:v0.0.14": GET https://example.com/users/sign_in: unexpected status code 200 OK: <!DOCTYPE html>
<HTML of the of the user login page>
...

The build stage is successful if we provide --no-push. It looks like what might be happening here is that as Kaniko tries logging into the registry at registry-example.com/v2/, and it at some point decides to visit example.com. That automatically redirects it to the user login page, which doesn't contain the expected data. (Our GitLab instance requires a login to view the / path on that site, but other public groups and repos are visible if you know their URLs).

I looked at the output of the env command in our runner, and that shows some entries that contain 'example.com', but it's not clear whether any of those are influencing Kaniko's behavior.

Is Kaniko looking for some data on the main page of our GitLab instance?

Expected behavior

The expected behavior is for Kaniko to log into the container registry, so it can push an image to it.

To Reproduce Steps to reproduce the behavior:

Create a self-hosted GitLab server installed via Omnibus.
Block anonymous viewing of the base path of the GitLab application.
Configure a separate domain to serve the registry API.
Use the following CI scripts in a new repo:

Additional Information

gcr.io/kaniko-project/executor:v1.22.0-debug (likely c7c1f8d3d464)

Dockerfile:

FROM ubuntu:mantic
RUN DEBIAN_FRONTEND=noninteractive apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y git yamllint

CMD /bin/bash

.gitlab-ci.yml:

---
build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.22.0-debug
    entrypoint: ["/bin/sh"]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" > /kaniko/.docker/config.json
    - cat /kaniko/.docker/config.json
    - echo "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
    - env
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
  rules:
    - if: $CI_COMMIT_TAG

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [ ]
Please check if this error is seen when you use `--cache` flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [ ]

Here are the env lines that match 'example.com':

CI_DEPENDENCY_PROXY_SERVER=example.com:443
CI_SERVER_HOST=example.com
CI_SERVER_URL=https://example.com
CI_COMPONENT_FQDN=example.com
CI_REGISTRY_IMAGE=registry-example.com:443/me/registry-test
CI_PIPELINE_URL=https://example.com/me/registry-test/-/pipelines/3612
CI_SERVER_FQDN=example.com
CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@example.com/me/registry-test.git
CI_API_GRAPHQL_URL=https://example.com/api/graphql
CI_REGISTRY=registry-example.com:443
CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX=example.com:443/me/dependency_proxy/containers
CI_API_V4_URL=https://example.com/api/v4
CI_SERVER_SHELL_SSH_HOST=example.com
CI_JOB_URL=https://example.com/me/registry-test/-/jobs/7576
CI_PROJECT_URL=https://example.com/me/registry-test
CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX=example.com:443/me/dependency_proxy/containers

I'm trying to compile a modified version of Kaniko for debugging purposes:

DOCKER_BUILDKIT=1 docker image build . -f deploy/Dockerfile --tag=kaniko

How can I build a -debug version / flavor so it includes /bin/sh, which is required by GitLab's CI runner?

I'm trying to compile a modified version of Kaniko for debugging purposes:
DOCKER_BUILDKIT=1 docker image build . -f deploy/Dockerfile --tag=kaniko
How can I build a -debug version / flavor so it includes /bin/sh, which is required by GitLab's CI runner?

I was able to add busybox with an extra Dockerfile:

FROM busybox:musl AS busybox

FROM kaniko:latest AS kaniko

ENV PATH="/bin:/kaniko"

COPY --from=busybox /bin /bin

Which allowed me to run my locally compiled version with GitLab. I'm still not sure why Kaniko is failing to log in though.

I added and used the following debug patch to Kaniko:

diff --git a/pkg/executor/push.go b/pkg/executor/push.go
index c95aecc3..393d345d 100644
--- a/pkg/executor/push.go
+++ b/pkg/executor/push.go
@@ -91,6 +91,7 @@ var (
 // push to every specified destination.
 func CheckPushPermissions(opts *config.KanikoOptions) error {
        targets := opts.Destinations
+       fmt.Printf("destinations: %s\n", opts.Destinations)
        // When no push and no push cache are set, we don't need to check permissions
        if opts.SkipPushPermissionCheck {
                targets = []string{}

Which prints the following, when adding a container name to the command, which should be necessary because we need more levels in the URL path:

$ /kaniko/executor --context "${CI_PROJECT_DIR}" --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination "${CI_REGISTRY_IMAGE}/foo:${CI_COMMIT_TAG}"
destinations: [registry-example.com:443/leaf-node/registry-test/foo:v0.0.19]
...
(same error seen originally)

The destination URL is correct. I get the same error with and without a the following in .gitlab-ci.yml:

    - mkdir -p /kaniko/.docker/
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" > /kaniko/.docker/config.json

I found a workaround for this issue: explicitly enter the registry URL so it's possible to leave off the port number :443.

      # working:
      --destination "registry-example.com/me/registry-test/my-container:${CI_COMMIT_TAG}"
      # not working:
      --destination "registry-example.com:443/me/registry-test/my-container:${CI_COMMIT_TAG}"

It may also be possible to configure GitLab to not include the port in the ${CI_REGISTRY_IMAGE} variable:

/etc/gitlab/gitlab.rb:

## comment out this line:
#gitlab_rails['registry_port'] = "443"

I haven't tested that yet, because our dev instance is currently facing another issue.

In any case, you'll still likely need to add something like /my-container to the image URL.

GoogleContainerTools / kaniko

"GET https://example.com/users/sign_in: unexpected status code 200 OK" when pushing to self-hosted GitLab with Kaniko #3146