Closed mjhoffman65 closed 4 years ago
I was able to fix this manually by updating the jenkins-docker-cfg
secret to:
{
"credHelpers": {
"<YOUR ECR HERE>": "ecr-login"
}
}
Would love to contribute the fix to this issue, but might need some help getting started
I was able to fix this manually by updating the
jenkins-docker-cfg
secret to:{ "credHelpers": { "<YOUR ECR HERE>": "ecr-login" } }
Would love to contribute the fix to this issue, but might need some help getting started if i need to change from ECR to Docker hub how can i do it
@deanesmith follow up
if i need to change from ECR to Docker hub how can i do it
I also have the same question as @mar-rih
if i need to change from ECR to Docker hub how can i do it
I also have the same question as @mar-rih
For a private dockerhub registry? It should just work with public images
I updated my jx/jenkins-docker-cfg/config.json
to
{
"credsStore": "ecr-login",
"credHelpers": {
"ecr.eu-central-1.amazonaws.com": "ecr-login"
}
}
and the problem is solved. Hope that help.
jx version
output
NAME VERSION
jx 2.0.702
jenkins x platform 2.0.1141
Kubernetes cluster v1.14.6-eks-5047ed
kubectl v1.15.3
helm client v2.14.3+g0e7f3b6
helm server v2.14.3+g0e7f3b6
git 2.23.0
Operating System Mac OS X 10.15 build 19A546d
I also updated my k8s secret, with content:
{
"credsStore": "ecr-login",
"credHelpers": {
"ecr.us-west-2.amazonaws.com": "ecr-login"
}
}
Even restarted Jenkins, but it did not help and I'm still getting error:
ONBUILD instructions: parsing ONBUILD instructions: processing base image (composer:1) for ONBUILD triggers: getting remote config: getting image: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fcomposer%3Apull&service=registry.docker.io: invoking docker-credential-ecr-login: exit status 1; output: 2019-09-10T12:40:53Z [ERROR] Error parsing the serverURL: https://index.docker.io, error: docker-credential-ecr-login can only be used with Amazon Elastic Container Registry.\ncredentials not found in native keychain"
jx version
output
NAME VERSION
jx 2.0.705
jenkins x platform 2.0.1119
Kubernetes cluster v1.13.10-eks-5ac0f1
kubectl v1.15.3
helm client v2.14.3+g0e7f3b6
helm server v2.14.3+g0e7f3b6
git 2.23.0
Operating System Mac OS X 10.14.6 build 18G95
I updated config.json content as
{
"credHelpers": {
"ecr.us-west-2.amazonaws.com": "ecr-login"
}
}
Removed "credsStore": "ecr-login",
, and that error dissapered, but I got a new one:
time="2019-09-10T13:16:52Z" level=fatal msg="build failed: build failed: building [redlotus-corp/cdp-web]: build artifact: unable to stream build output: Error parsing reference: \"composer:1 AS build-env\" is not a valid repository/tag: invalid reference format"
@stepanselyuk I also updated all of the Jenkins plugins (with a restart), except ones that can't be upgrade.
Last error seems caused by an older version of Docker installed on the nodes. I use amiFamily: Ubuntu1804
in eksctl
config.
docker version
Client:
Version: 17.03.2-ce
API version: 1.27
Go version: go1.9.4
Git commit: f5ec1e2
Built: Thu Jul 11 12:25:38 2019
OS/Arch: linux/amd64
Multi-stage builds were introduced in 17.06 https://blog.docker.com/2017/07/multi-stage-builds/
Updated docker version on the nodes and re-run build, next I've got an error related to push to ECR. Fixed by using this configuration:
{
"credHelpers": {
"{ACCOUNT_ID}.dkr.ecr.{ECR_REGION}.amazonaws.com": "ecr-login"
}
}
"credsStore": "ecr-login",
not needed cause it make these credentials use by default to any registry.
If you need to use a private registry (e.g. private Docker Hub), you can use a configuration like this:
{
"auths": {
"index.docker.io/v1": {"auth": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" }
},
"credHelpers": {
"{ACCOUNT_ID}.dkr.ecr.{ECR_REGION}.amazonaws.com": "ecr-login"
}
}
Where is XXXX.. is your token.
On Ubuntu Nodes you can update your Docker version by executing commands below (from Ubuntu user), and it's not recommended for production, cause you need to have more control what to do.
sudo apt-get remove -y docker docker-engine docker.io containerd runc
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
More information on this topic: https://docs.docker.com/install/linux/docker-ce/ubuntu/
It is possible that you will need to update Jenkins plugins again, but I'm not sure what caused that in my case, cause first I've upgraded EKS cluster to 1.14 version and then upgraded Docker on Nodes.
I have this precise problem for a customer and I'm unable to provide them the EKS-backed CI/CD solution until this is patched.
I've followed the above recommendations to change the config.json secret, which has not worked for me at all.
I have also destroyed my EKS cluster and rebuild it and re-installed JX, and I have the exact same outcome.
I have updated my jx CLI (brew upgrade jx) and also pulled down the latest cloud-environments repository and such when prompted during cluster and JX installations, so I'm confident this hasn't been patched yet. If there's some kind of canary build I can use/test with I'd be more than happy to give it a spin! 👍 :)
Is someone actively working on this fix? I'd like to be able to update my customer on an ETA to see if they want to wait for this EKS fix or come up with a workaround.
Thanks! and keep up the great work 👍
/cc @martin-patsov
@dragoonis for me this configuration works:
{
"credHelpers": {
"{ACCOUNT_ID}.dkr.ecr.{ECR_REGION}.amazonaws.com": "ecr-login"
}
}
Please be sure you properly encoded this content to base64 and placed in jenkins-docker-cfg
secret.
cat <<EOF | base64
{
"credHelpers": {
"{ACCOUNT_ID}.dkr.ecr.{ECR_REGION}.amazonaws.com": "ecr-login"
}
}
EOF
What exactly error do you have in the logs?
@stepanselyuk I had not put in the ACCOUNT_ID part before. I will try that.
However, I still believe that when making requests to Docker Hub (anonymously ofcourse) to pull in official base images, that it will say docker-credential-ecr-login can only be used with Amazon Elastic Container Registry
meaning it's trying to send ecr-login to Docker Hub.
I will get back to you with trying the ACCOUNT_ID
@dragoonis please be sure you don't use "credsStore": "ecr-login",
cause that defaults all registries hosts to be used with ecr-login
, and ecr-login
can be used with ECR only as I see in the error message.
If you don't use ACCOUNT_ID, you won't be able to push images into ECR.
It's worked! partially..
TLDR; Docker pull (base image) from Docker Hub is now working, but only for 1 of 2 images. Docker push to ECR is working.
The one that works (company-api-nginx) manages to pull from Docker Hub and be built and pushed to ECR. Second one (company-api-php-fpm) fails to pull from Docker Hub
- company-api-nginx: Not found. Building
- company-api-php-fpm: time="2019-09-23T20:35:36Z" level=warning msg="error checking cache, caching may not work as expected: getting hash for artifact company-api-php-fpm: getting dependencies for company-api-php-fpm: expanding ONBUILD instructions: parsing ONBUILD instructions: processing base image (php:7-fpm-alpine) for ONBUILD triggers: getting remote config: getting image: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fphp%3Apull&service=registry.docker.io: invoking docker-credential-ecr-login: exit status 1; output: 2019-09-23T20:35:35Z [ERROR] Error parsing the serverURL: https://index.docker.io, error: docker-credential-ecr-login can only be used with Amazon Elastic Container Registry.\ncredentials not found in native keychain"
Successfully tagged xxxxxxx.dkr.ecr.eu-west-2.amazonaws.com/company/company-api-nginx:0.0.0-SNAPSHOT-PR-628-3
The push refers to repository [xxxxxxxx.dkr.ecr.eu-west-2.amazonaws.com/company/company-api-nginx]
e929d9605539: Preparing
72c96a498309: Preparing
a0a699786342: Preparing
8b5b6870672a: Preparing
03fb8dd8a4de: Preparing
0665fde51bac: Preparing
3e76d2df1790: Preparing
03901b4a2ea8: Preparing
0665fde51bac: Waiting
3e76d2df1790: Waiting
03901b4a2ea8: Waiting
72c96a498309: Pushed
e929d9605539: Pushed
a0a699786342: Pushed
3e76d2df1790: Layer already exists
03901b4a2ea8: Layer already exists
03fb8dd8a4de: Pushed
0665fde51bac: Pushed
8b5b6870672a: Pushed
0.0.0-SNAPSHOT-PR-628-3: digest: sha256:00a6f1813538ceb9e6296c8741d987e48d010b4b5940b05e80c96b53ceb2a36c size: 1986
Building [company-api-php-fpm]...
Sending build context to Docker daemon
time="2019-09-23T20:35:57Z" level=fatal msg="build failed: build failed: building [company-api-php-fpm]: build artifact: docker build: error during connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.38/build?buildargs=null&cachefrom=null&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=docker%2Fphp-fpm%2FDockerfile&labels=null&memory=0&memswap=0&networkmode=&rm=0&shmsize=0&t=xxxxxxxxx.dkr.ecr.eu-west-2.amazonaws.com%2Fcompany%2Fcompany-api-php-fpm%3A0.0.0-SNAPSHOT-PR-628-3&target=&ulimits=null&version=: creating docker context: getting relative tar paths: expanding ONBUILD instructions: parsing ONBUILD instructions: processing base image (php:7-fpm-alpine) for ONBUILD triggers: getting remote config: getting image: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fphp%3Apull&service=registry.docker.io: invoking docker-credential-ecr-login: exit status 1; output: 2019-09-23T20:35:57Z [ERROR] Error parsing the serverURL: https://index.docker.io, error: docker-credential-ecr-login can only be used with Amazon Elastic Container Registry.\ncredentials not found in native keychain"
@dragoonis if it worked partially, I think it is possible that erroneous build was affected by old configuration. Please try again (the PR rebuild), and if all the same, I think better to check actual Docker config on each node. Maybe it worth also to restart Jenkins.
We switched to
{ "credHelpers": { "<YOUR ECR HERE>": "ecr-login" } }
and worked like charm, guessing that the node expires the token after several days...
I was fixed this problem by:
jenkins-docker-cfg
secret:
kubectl get secret jenkins-docker-cfg -o yaml
I got the result as follow:
apiVersion: v1
data:
config.json: <Secret DATA>
kind: Secret
metadata:
...
Then base64 encode the <Secret DATA>
use echo '<Secret DATA> '' | base64 --decode
, and i saw the secret result was:
{
"credsStore": "ecr-login"
}
jenkins-docker-cfg
by using kubectl delete secret jenkins-docker-cfg
;config.json
with:
{
"credHelpers": {
"{ACCOUNT_ID}.dkr.ecr.{ECR_REGION}.amazonaws.com": "ecr-login"
}
}
jenkins-docker-cfg
secret by using:
kubectl create secret generic jenkins-docker-cfg --from-file=./config.json
I using AWS EKS and jenkins-x, with jx version:
NAME VERSION
jx 2.0.861
jenkins x platform 2.0.1441
Kubernetes cluster v1.14.6-eks-5047ed
kubectl v1.12.7
helm client Client: v2.14.1+g5270352
git 2.14.5
Operating System Unknown Linux distribution Linux version 4.14.114-82.97.amzn1.x86_64 (mockbuild@gobi-build-64003) (gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC)) #1 SMP Sun Apr 28 07:27:43 UTC 2019
thanks @yuwenhui
$ ERC=$(echo '{ "credsStore": {"999999999999.dkr.ecr.ap-southeast-2.amazonaws.com": "ecr-login"} }'| base64 -w 0) $ kubectl patch secret jenkins-docker-cfg -p="{\"data\": { \"config.json\": \"$ERC\"}}"
I'm glad this is now patched.
Summary
Build fails while attempting to pull images from Docker Hub.
Steps to reproduce the behavior
Run any build with a Dockerfile that has a base image from Docker Hub (or anywhere besides ECR) in an EKS 1.13 cluster with ECR enabled for classic jenkins.
Expected behavior
The docker build will pull the base image from Docker Hub without error when running the
skaffold build
command.Actual behavior
Build fails when building the docker images with skaffold with an error stating docker-credential-ecr-login can only be used with ECR.
Jx version
The output of
jx version
is:Jenkins type
Kubernetes cluster
EKS 1.13 created with eksctl and ECR.
Operating system / Environment
Ubuntu 18.04.2 LTS