search

GoogleContainerTools / kaniko

Build Container Images In Kubernetes
Apache License 2.0
14.61k stars 1.42k forks source link

On 1.17.0, instance metadata ECR credentials authorization on EC2 doesn't work anymore #2808

Open rdbisme opened 11 months ago

rdbisme commented 11 months ago

Actual behavior We have the following line in our CI script:

This enables kaniko to authorize the push to ECR registry using the EC2 machine instance metadata it's running on. This stopped working with 1.17.0 with the following error:

panic: failed to get shared config profile, <redacted>
goroutine 1 [running]:
github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api.DefaultClientFactory.NewClientFromRegion({}, {0xc00015945d?, 0x0?})
    /src/vendor/github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api/factory.go:84 +0x219
github.com/awslabs/amazon-ecr-credential-helper/ecr-login.ECRHelper.Get({{0x882240?, 0xae6780?}, 0xab0a40?}, {0xc000132540, 0x2c})
    /src/vendor/github.com/awslabs/amazon-ecr-credential-helper/ecr-login/ecr.go:101 +0x113
github.com/docker/docker-credential-helpers/credentials.Get({0x881e50, 0xc00011a0d8}, {0x87d840?, 0xc000110020?}, {0x87d7c0, 0xc000110028})
    /src/vendor/github.com/docker/docker-credential-helpers/credentials/credentials.go:130 +0x1fa
github.com/docker/docker-credential-helpers/credentials.HandleCommand({0x881e50?, 0xc00011a0d8?}, {0x7ffedf9c87be?, 0xc00018bec0?}, {0x87d840?, 0xc000110020?}, {0x87d7c0?, 0xc000110028?})
    /src/vendor/github.com/docker/docker-credential-helpers/credentials/credentials.go:73 +0x85
github.com/docker/docker-credential-helpers/credentials.Serve({0x881e50?, 0xc00011a0d8?})
    /src/vendor/github.com/docker/docker-credential-helpers/credentials/credentials.go:58 +0xee
main.main()
    /src/vendor/github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login/main.go:44 +0x154
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "[MASKED]/xxxx/xxx": resolving authorization for [MASKED] failed: error getting credentials - err: exit status 2, out: ``

Expected behavior Authorization should keep working.

To Reproduce Steps to reproduce the behavior:

  1. ...
  2. ...

Additional Information

csm-kb commented 11 months ago

Can confirm the same! via executor:debug as of 10 minutes ago:

$ echo "{\"credsStore\":\"ecr-login\",\"credHelpers\":{\"${DOCKER_REGISTRY}\":\"ecr-login\"}}" > /kaniko/.docker/config.json
$ echo "Creating ${CI_env} build for ${DOCKER_REGISTRY}/${APP_NAME}:${IMAGE_TAG}"
Creating staging build for [MASKED].dkr.ecr.[MASKED].amazonaws.com/[MASKED]:v0.33.7
$ export AWS_PROFILE=default
$ /kaniko/executor --context ${CI_PROJECT_DIR} --dockerfile ${CI_PROJECT_DIR}/Dockerfile --build-arg "BUILD_APP_ENV=${CI_env}" --destination "${DOCKER_REGISTRY}/${APP_NAME}:${IMAGE_TAG}" --cache=true --cache-repo "${DOCKER_REGISTRY}/${APP_NAME}" --cache-ttl ${CACHE_TTL}
panic: failed to get shared config profile, default
goroutine 1 [running]:
github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api.DefaultClientFactory.NewClientFromRegion({}, {0xc00002d8dd?, 0x0?})
    /src/vendor/github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api/factory.go:84 +0x[21](https://gitlab.com/[MASKED]/-/jobs/5335232340#L21)9
github.com/awslabs/amazon-ecr-credential-helper/ecr-login.ECRHelper.Get({{0x88[22](https://gitlab.com/[MASKED]/-/jobs/5335232340#L22)40?, 0xae6780?}, 0xab0a40?}, {0xc000026b70, 0x2c})
    /src/vendor/github.com/awslabs/amazon-ecr-credential-helper/ecr-login/ecr.go:101 +0x113
github.com/docker/docker-credential-helpers/credentials.Get({0x881e50, 0xc0000100f0}, {0x87d840?, 0xc000068028?}, {0x87d7c0, 0xc000068030})
    /src/vendor/github.com/docker/docker-credential-helpers/credentials/credentials.go:130 +0x1fa
github.com/docker/docker-credential-helpers/credentials.HandleCommand({0x881e50?, 0xc0000100f0?}, {0x7ffc5[24](https://gitlab.com/[MASKED]/-/jobs/5335232340#L24)af69f?, 0xc000161ec0?}, {0x87d840?, 0xc0000680[28](https://gitlab.com/[MASKED]/-/jobs/5335232340#L28)?}, {0x87d7c0?, 0xc0000680[30](https://gitlab.com/[MASKED]/-/jobs/5335232340#L30)?})
    /src/vendor/github.com/docker/docker-credential-helpers/credentials/credentials.go:73 +0x85
github.com/docker/docker-credential-helpers/credentials.Serve({0x881e50?, 0xc0000100f0?})
    /src/vendor/github.com/docker/docker-credential-helpers/credentials/credentials.go:58 +0xee
main.main()
    /src/vendor/github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login/main.go:44 +0x154
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "[MASKED].dkr.ecr.[MASKED].amazonaws.com/[MASKED]:v0.[33](https://gitlab.com/[MASKED]/-/jobs/5335232340#L33).7": resolving authorization for [MASKED].dkr.ecr.[MASKED].amazonaws.com failed: error getting credentials - err: exit status 2, out: ``
aaron-prindle commented 10 months ago

@rdbisme and @csm-kb - thank you for flagging the issue here. I haven't had a chance to investigate this regression more deeply. Below is a list of all of the changes made from v1.16.0 - v1.17.0, likely one of these changes caused this regression IIUC. From this list it seems that this would likely be related to one of the updated deps, please add additional information/investigation if anyone in the thread here has a sense of what the root cause might be.

Docs, Test, and CI/CD Updates:

Updates and Refactors: