Cannot push images to ECR

vfarcic commented 4 years ago

Summary

After setting up Jenkins X in EKS using jx boot, the permissions to push images to ECR are not correct.

The user used to create the cluster has full admin rights.

Steps to reproduce the behavior

cat jx-requirements.yml

autoUpdate:
  enabled: false
  schedule: ""
bootConfigURL: https://github.com/jenkins-x/jenkins-x-boot-config
cluster:
  clusterName: jx-eks
  devEnvApprovers:
  - vfarcic
  environmentGitOwner: vfarcic
  gitKind: github
  gitName: github
  gitServer: https://github.com
  namespace: jx
  provider: eks
  region: us-east-2
  registry: 036548781187.dkr.ecr.us-east-2.amazonaws.com
environments:
- ingress:
    domain: 3.12.183.11.nip.io
    externalDNS: false
    namespaceSubDomain: -jx.
    tls:
      email: ""
      enabled: false
      production: false
  key: dev
  repository: environment-jx-eks-dev
- ingress:
    domain: ""
    externalDNS: false
    namespaceSubDomain: ""
    tls:
      email: ""
      enabled: false
      production: false
  key: staging
  repository: environment-jx-eks-staging
- ingress:
    domain: ""
    externalDNS: false
    namespaceSubDomain: ""
    tls:
      email: ""
      enabled: false
      production: false
  key: production
  repository: environment-jx-eks-production
gitops: true
ingress:
  domain: 3.12.183.11.nip.io
  externalDNS: false
  namespaceSubDomain: -jx.
  tls:
    email: ""
    enabled: false
    production: false
kaniko: true
repository: nexus
secretStorage: vault
storage:
  backup:
    enabled: false
    url: ""
  logs:
    enabled: true
    url: s3://jx-eks-logs-eca3d4b9-f83c-48ba-8af4-bf308210b4e5
  reports:
    enabled: true
    url: s3://jx-eks-reports-3a31e691-79e9-4c3e-b66d-22d1b68e20f5
  repository:
    enabled: true
    url: s3://jx-eks-repository-3a726167-dcec-4b32-9c48-04e1f2670cf7
vault:
  aws:
    autoCreate: true
    iamUserName: jx-boot
  disableURLDiscovery: true
velero:
  schedule: ""
  ttl: ""
versionStream:
  ref: v1.0.348
  url: https://github.com/jenkins-x/jenkins-x-versions.git
webhook: prow

jx create quickstart \
    --filter golang-http

jx get activity \
    --filter jx-boot/master \
    --watch

jx get build logs

. . . 
Showing logs for build vfarcic-jx-boot-master-r565f-1 stage from-build-pack and container step-build-container-build
INFO[0000] No base image, nothing to extract            
INFO[0000] cmd: EXPOSE                                  
INFO[0000] Adding exposed port: 8080/tcp                
INFO[0000] Using files from context: [/workspace/source/bin] 
INFO[0000] Skipping unpacking as no commands require it. 
INFO[0000] Taking snapshot of full filesystem...        
INFO[0000] EXPOSE 8080                                  
INFO[0000] cmd: EXPOSE                                  
INFO[0000] Adding exposed port: 8080/tcp                
INFO[0000] No files changed in this command, skipping snapshotting. 
INFO[0000] ENTRYPOINT ["/**-boot"]                      
INFO[0000] No files changed in this command, skipping snapshotting. 
INFO[0000] Using files from context: [/workspace/source/bin] 
INFO[0000] COPY ./bin/ /                                
INFO[0000] Taking snapshot of files...                  
error pushing image: failed to push to destination 036548781187.dkr.ecr.us-east-2.amazonaws.com/*******/**-boot:0.0.1: DENIED: User: arn:aws:sts::036548781187:assumed-role/eksctl-**-eks-nodegroup-ng-c9e68b-NodeInstanceRole-13I64VIH6C65T/i-0187fadb40469c739 is not authorized to perform: ecr:InitiateLayerUpload on resource: arn:aws:ecr:us-east-2:036548781187:repository/*******/**-boot

Pipeline failed on stage 'from-build-pack' : container 'step-build-container-build'. The execution of the pipeline has stopped.

Expected behavior

The pipeline works

Actual behavior

The pipeline does not work

Jx version

The output of jx version is:

NAME               VERSION
jx                 2.0.1192
Kubernetes cluster v1.14.9-eks-c0eccc
kubectl            v1.17.0
helm client        Client: v2.14.3+g0e7f3b6
git                2.20.1 (Apple Git-117)
Operating System   Mac OS X 10.14.6 build 18G2022

Jenkins type

[X] Serverless Jenkins X Pipelines (Tekton + Prow)
[ ] Classic Jenkins

Kubernetes cluster

EKS

Operating system / Environment

macOS

rawlingsj commented 4 years ago

@vfarcic as a test could you try and edit the golang pod template config map (kubectl edit cm jenkins-x-pod-template-go), remove the DOCKER_CONFIG env var and trigger another pipeline to see if that is causing the problem?

I noticed a similar problem when using Googles workload identity

daveconde commented 4 years ago

When jx create quickstart or jx import is run a repository is created in the ECR. See: https://github.com/jenkins-x/jx/blob/master/pkg/cmd/importcmd/import.go#L1186 Was there anything in the logs around this?

dgozalo commented 4 years ago

It appears that it's using the k8s worker node role instead of the tekton-bot ServiceAccount attached role.

How did you create the cluster? Did you boot with the --terraform flag?

If so, you'll need to add enough permissions to the eksctl-jx-eks-nodegroup-ng-c9e68b-NodeInstanceRole-13I64VIH6C65T role.

vfarcic commented 4 years ago

My bad... I expected jx boot to add the policies it needs (it's been doing that does other cases) or to fail the validation stage. If that's not the intention, the following can be put to the docs for those using eksctl.

CLUSTER_NAME=[...] # Replace `[...]` with the name of the cluster

IAM_ROLE=$(aws iam list-roles \
    | jq -r ".Roles[] \
    | select(.RoleName \
    | startswith(\"eksctl-$CLUSTER_NAME-nodegroup\")) \
    .RoleName")

aws iam attach-role-policy \
    --role-name $IAM_ROLE \
    --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess

mb-lyrical commented 4 years ago

there is a similar issue with AKS repos too. But this was solved with a new format for the docker credentials file config.json using unencoded username/password fields v's the auth element.

jenkins-x / jx