search

GoogleContainerTools / skaffold

Easy and Repeatable Kubernetes Development
https://skaffold.dev/
Apache License 2.0
15.04k stars 1.62k forks source link

Error after building (Kaniko) and pushing an image to a Gitlab private repository #6352

Closed toyi closed 2 years ago

toyi commented 3 years ago

Hello,

New to Skaffold here, I'm having a hard time building an image for a private Gitlab repository using Kaniko. Since it works well using Kaniko alone, I presume the problem is from Skaffold. The image is indeed built and pushed, but an error still appears at the end. This error could be related to https://github.com/GoogleContainerTools/skaffold/issues/4287 (almost the same error message).

Expected behavior

There should be no error after the image is pushed.

Actual behavior

After the image is pushed, there is an error (preventing ci/cd from succeeding). GET https://gitlab.com/jwt/auth?scope=repository%3Atoyi%2Fskaffold%3Apull&service=container_registry: DENIED: access forbidden

Information

apiVersion: skaffold/v2beta19
kind: Config
metadata:
  name: skaffold
build:
  cluster:
    serviceAccount: default
    randomDockerConfigSecret: true
    dockerConfig:
      path: .dockerconfigjson

  artifacts:
    - image: registry.gitlab.com/toyi/skaffold
      context: .
      kaniko:
        image: gcr.io/kaniko-project/executor:debug
        dockerfile: Dockerfile

Dockerfile: 

FROM alpine:latest

.dockerconfigjson:

{
  "auths":{
    "https://registry.gitlab.com":{
      "username":"toyi",
      "password":"password",
      "email":"myemail@mydomain.com",
      "auth":"dG95aTpwYXNzd29yZA=="
    }
  }
}

Steps to reproduce the behavior

  1. Make a private Gitlab repository
  2. Create a deploy token (or use your personal credentials, I have the same error both ways)
  3. skaffold build

This error only appears when the machine executing skaffold build doesn't have a direct access to the private Gitlab repository. What I mean is that I don't have this error if I actually have the credentials saved in my own ~/.docker/config.json file. That's why I first encountered this error in a CI/CD pipeline, since that's the purpose of Kaniko (not using Docker).

Back to my local machine, I am able to reproduce this problem by running docker logout registry.gitlab.com. If I login again (docker login registry.gitlab.com), everything is fixed.

Maybe I'm missing something that's entirely possible, as I said I'm pretty new to the Kubernetes world, but since Skaffold (and Kaniko) both have the credentials through the .dockerconfigjson, and since everything is built AND pushed, I presume this error shouldn't be there ?

I noticed the pull part of the request GET https://gitlab.com/jwt/auth?scope=repository%3Atoyi%2Fskaffold%3Apull&service=container_registry: DENIED: access forbidden. However, even if I don't really understand why a pull is attempted, I assure you the credentials used here can pull images from this repository. For the purpose of this issue, I disabled 2FA and used my simple username / password. The result is the same with 2FA enabled or with a deploy token.

Standard logs

Generating tags...
 - registry.gitlab.com/toyi/skaffold -> registry.gitlab.com/toyi/skaffold:cf3db33-dirty
Checking cache...
 - registry.gitlab.com/toyi/skaffold: Not found. Building
Starting build...
Creating docker config secret [docker-cfg4c250433-f496-11eb-9945-b42e99a9275c]...
Building [registry.gitlab.com/toyi/skaffold]...
INFO[0001] Retrieving image manifest alpine:latest      
INFO[0001] Retrieving image alpine:latest from registry index.docker.io 
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest alpine:latest      
INFO[0002] Returning cached image manifest              
INFO[0002] Executing 0 build triggers                   
INFO[0002] Skipping unpacking as no commands require it. 
INFO[0002] Pushing image to registry.gitlab.com/toyi/skaffold:cf3db33-dirty 
INFO[0008] Pushed image to 1 destinations               
GET https://gitlab.com/jwt/auth?scope=repository%3Atoyi%2Fskaffold%3Apull&service=container_registry: DENIED: access forbidden

With -vdebug:

INFO[0000] Skaffold &{Version:v1.28.1 ConfigVersion:skaffold/v2beta19 GitVersion: GitCommit:7b855e136dc0f9cc5544ed2808c9d5feb767118c BuildDate:2021-07-20T17:49:44Z GoVersion:go1.14.14 Compiler:gc Platform:linux/amd64 User:} 
INFO[0000] Loaded Skaffold defaults from "/home/toyi/.skaffold/config" 
DEBU[0000] parsed 1 configs from configuration file /home/toyi/Projects/toyi/skaffold/skaffold.yaml 
DEBU[0000] Defaulting deploy type to kubectl            
INFO[0000] Using kubectl context: k3d-skaffold          
DEBU[0000] Running command: [minikube version --output=json] 
DEBU[0000] setting Docker user agent to skaffold-v1.28.1 
DEBU[0000] Using builder: cluster                       
INFO[0000] build concurrency first set to 0 parsed from *cluster.Builder[0] 
INFO[0000] final build concurrency value is 0           
Generating tags...
 - registry.gitlab.com/toyi/skaffold -> DEBU[0000] Running command: [git describe --tags --always] 
DEBU[0000] Command output: [cf3db33
]                   
DEBU[0000] Running command: [git status . --porcelain]  
DEBU[0000] Command output: [?? .idea/vcs.xml
]          
registry.gitlab.com/toyi/skaffold:cf3db33-dirty
INFO[0000] Tags generated in 2.786951ms                 
Checking cache...
DEBU[0001] Found dependencies for dockerfile: []        
DEBU[0001] Could not import artifact from Docker, building instead (import of missing images disabled) 
 - registry.gitlab.com/toyi/skaffold: Not found. Building
INFO[0001] Cache check completed in 1.171 second        
Starting build...
Creating docker config secret [docker-cfgbe237edc-f497-11eb-817b-e663c77070c0]...
DEBU[0001] getting client config for kubeContext: ``    
Building [registry.gitlab.com/toyi/skaffold]...
DEBU[0001] getting client config for kubeContext: ``    
INFO[0001] Waiting for kaniko-8lh78 to be initialized   
DEBU[0002] Running command: [kubectl --context k3d-skaffold exec -i kaniko-8lh78 -c kaniko-init-container -n default -- tar -xf - -C /kaniko/buildcontext] 
DEBU[0003] Running command: [kubectl --context k3d-skaffold exec kaniko-8lh78 -c kaniko-init-container -n default -- touch /tmp/complete] 
DEBU[0003] Command output: []                           
INFO[0003] Waiting for kaniko-8lh78 to be complete      
DEBU[0003] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-8lh78" is waiting to start: PodInitializing 
DEBU[0004] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-8lh78" is waiting to start: PodInitializing 
DEBU[0005] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-8lh78" is waiting to start: PodInitializing 
INFO[0001] Retrieving image manifest alpine:latest      
INFO[0001] Retrieving image alpine:latest from registry index.docker.io 
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest alpine:latest      
INFO[0002] Returning cached image manifest              
INFO[0002] Executing 0 build triggers                   
INFO[0002] Skipping unpacking as no commands require it. 
INFO[0002] Pushing image to registry.gitlab.com/toyi/skaffold:cf3db33-dirty 
INFO[0007] Pushed image to 1 destinations               
DEBU[0014] Running command: [tput colors]               
DEBU[0014] Command output: [256
]                       
GET https://gitlab.com/jwt/auth?scope=repository%3Atoyi%2Fskaffold%3Apull&service=container_registry: DENIED: access forbidden
DEBU[0014] exporting metrics                            
DEBU[0015] metrics uploading complete in 574.157616ms

It has been pushed: image

toyi commented 3 years ago

I just found https://github.com/GoogleContainerTools/skaffold/issues/5571 which seems to be related. Without even docker installed, if I just put the credentials in ~/.docker/config.json, it works.

However, I still think it should not be the case.

@tejal29 Just tagging you, since you are already working on this topic in https://github.com/GoogleContainerTools/skaffold/issues/5571, you might be interested by this issue as well.

aaron-prindle commented 3 years ago

Friendly ping @tejal29, are there any plans to prioritize #5571 into upcoming sprints/milestones? Seems this core issue here is with the docker creds skaffold uses with kaniko and looks to be a dupe of #5571.

cslovell commented 3 years ago

I had this issue--basically, we are moving away from Docker Desktop given the non-open source nature and have removed all traces of docker from our systems.

Initially we had a kaniko-secret (below) loaded into k8s and referenced it in every way possible, but ultimately nothing worked--until we created a ~/.docker/config.json file with the same snippet: 

{"auths":{"registry.gitlab.com":{"username":"me","password":"passyMcPassface","email":"email"}}}

I feel I'm missing something - but my sense is that it shouldn't need the local config.json file since it's all built in cluster using kaniko anyway...

aaron-prindle commented 3 years ago

Currently this work is not scoped for any upcoming sprints/milestones. Moving to priority p3

jr200 commented 3 years ago

Also got hit by this and found this github issue after several hours of confusion. Perhaps requires documentation if theres no immediate fix?

My setup is:

apiVersion: skaffold/v2beta25
kind: Config

build:
  artifacts:
  - image: cloud.canister.io:5000/dummy/dummyimage
    context: backend
    kaniko:
      dockerfile: Dockerfile
      image: gcr.io/kaniko-project/executor
      verbosity: info
  cluster:
    namespace: kaniko
    dockerConfig:
      secretName: cloud.canister.io-push

If I run skaffold build, the image builds and at the very last step I get:

INFO[0019] Pushing image to cloud.canister.io:5000/dummy/dummyimage:latest 
INFO[0023] Pushed image to 1 destinations               
no token in bearer response:
{"errors":[{"type":"Unauthorized","message":"Not authorized to access this resource."}]}

If I create a local file ~/.docker/config.json on my mac, with the same credentials as above (i.e., push credentials) the build step succeeds.

gsquared94 commented 2 years ago

I think that the offending code is here: https://github.com/GoogleContainerTools/skaffold/blob/9e8762f27809dfe593ecf23a1f10cdc18ccb1c24/pkg/skaffold/build/cluster/kaniko.go#L94 which tries to fetch the remote image digest as the last step after the image has been built.

Adding the ~/.docker/config.json file with credentials works since we're adding a credential helper to read from that directory always (irrespective of if docker binary is installed or not) here: https://github.com/GoogleContainerTools/skaffold/blob/9e8762f27809dfe593ecf23a1f10cdc18ccb1c24/pkg/skaffold/docker/auth.go#L38-L52

Skaffold needs to be able to read the image digest and as such it needs these credentials but that IMHO could be somewhat conflicting with why users are using kaniko in the first place. I'll bring this up in our triage meeting.

tejal29 commented 2 years ago

@gsquared94 Kaniko does not return the digest of the built image and hence in order for skaffold to replace the image in deploy manifests skaffold needs to fetch the remote image build.

The other option is to add json output to kaniko which will return the digest of the built image. Since kaniko is in maintenance mode I don't see it happen.

Closing this as Wont-fix

Please re-open if you have any follow up questions.

jagwire16 commented 2 years ago

I had the same issue and am using GitLab pipelines to build the images via Skaffold & Kaniko. This is the fix that worked for me in .gitlab-ci.yaml.

stages:
  - deploy

# Run Skaffold
skaffold-run:
  stage: deploy
  image:
    name: registry.gitlab.com/redacted-org/redacted-project/skaffold:v1.38.0
    entrypoint: ['']
  script:
    # For Kaniko
    - echo "${DOCKER_CONFIG_JSON}" > .dockerconfigjson
    # Required due to https://github.com/GoogleContainerTools/skaffold/issues/6352
    # For Skaffold
    - mkdir -p ~/.docker
    - echo "${DOCKER_CONFIG_JSON}" > ~/.docker/config.json
    - skaffold run

Note: DOCKER_CONFIG_JSON contains the full file contents from ~/.docker/config.json