search

jx3-gitops-repositories / jx3-terraform-eks

Jenkins X 3.x Infrastructure Git Template for Terraform and EKS for managing cloud resources
Apache License 2.0
9 stars 41 forks source link

Registry Failed #24

Closed maiakhoa closed 3 years ago

maiakhoa commented 3 years ago

does anyone face this issue while running pipeline?

loading config with AWS region: 'us-east-1'
--
Let's ensure that we have an ECR repository for the image 
error: failed to lazy create the ECR registry for : failed to check for repository with registry ID : operation error ECR: DescribeRepositories, failed to sign request: failed to retrieve credentials: failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, https response error StatusCode: 400, RequestID: 51e765ba-59c0-4a4d-b3fa-e9b1832f8512, api error InvalidIdentityToken: No OpenIDConnect provider found in your account for
ankitm123 commented 3 years ago

So this issue is related to not having an open ID connect provider for ur cluster. Normally when you create an eks cluster, it should get created alongside the cluster. Once that OIDC provider is up, it's used by the pods to assume iam roles associated with their service account. See: https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html

maiakhoa commented 3 years ago

thank @ankitm123

I can pass that step. But it stuck at build image for both pipelines and Jenkins Server

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: getting tag for destination: repository can only contain the runesabcdefghijklmnopqrstuvwxyz0123456789_-./``

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "": unsupported status code 401; body: Not Authorized

do you have any idea that help me to fix that?

ankitm123 commented 3 years ago

do u have push access to ecr repo and also create it? Does the repo exist? This may be related: https://github.com/tektoncd/pipeline/issues/992

maiakhoa commented 3 years ago

it is existed, I used that for a long time ago with Jenkins X 2

ankitm123 commented 3 years ago

can you paste your pipeline step where you are building/pushing images to ecr?

maiakhoa commented 3 years ago 
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  creationTimestamp: null
  name: release
spec:
  pipelineSpec:
    tasks:
      - name: from-build-pack
        resources: {}
        taskSpec:
          metadata: {}
          stepTemplate:
            env:
              - name: _JAVA_OPTIONS
                value: -XX:+UnlockExperimentalVMOptions -Dsun.zip.disableMemoryMapping=true
                  -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4
                  -XX:AdaptiveSizePolicyWeight=90 -Xms10m -Xmx1024m
              - name: MAVEN_OPTS
                value: -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.httpconnectionManager.ttlSeconds=25 -Dmaven.wagon.http.retryHandler.count=3
            image: uses:jenkins-x/jx3-pipeline-catalog/tasks/maven-java11/release.yaml@versionStream
            name: ""
            resources:
              requests:
                cpu: 400m
                memory: 1024Mi
            volumeMounts:
              - mountPath: /root/.m2/
                name: maven-settings
              - mountPath: /root/.gnupg
                name: release-gpg
            workingDir: /workspace/source
          steps:
            - image: uses:jenkins-x/jx3-pipeline-catalog/tasks/git-clone/git-clone.yaml@versionStream
              name: ""
              resources: {}
            - name: next-version
              resources: {}
            - name: jx-variables
              resources: {}
            - name: build-mvn-deploy
              resources: {}
            - name: check-registry
              resources: {}
            - name: build-container-build
              resources: {}
          volumes:
            - name: maven-settings
              secret:
                secretName: jenkins-maven-settings
            - name: release-gpg
              secret:
                optional: true
                secretName: jenkins-release-gpg
  podTemplate: {}
  serviceAccountName: tekton-bot
  timeout: 12h0m0s
status: {}

This is the one I copied from quickstart template

maiakhoa commented 3 years ago

I'm also facing the same issue when using Kaniko to build image from Jenkins Server. Do you have any solution to enable Kanito support for the Jenkins X 3?

ankitm123 commented 3 years ago

So I guess this step is where ur pipeline fails: build-container-build? Can you print out the environment variables at that step, specifically PUSH_CONTAINER_REGISTRY and DOCKER_REGISTRY_ORG. I believe they are not correctly set. FYI, this is the place where this pipeline catalog comes from: https://github.com/jenkins-x/jx3-pipeline-catalog/blob/master/tasks/maven-java11/pullrequest.yaml#L61-L68

ankitm123 commented 3 years ago

Do you have any solution to enable Kanito support for the Jenkins X 3?

There is no extra step to enable kaniko for JX3. It's just a binary inside a docker container. The build-container-build step already uses kaniko. I will make a PR to update the version of kaniko from 1.3.0 to 1.6.0.

maiakhoa commented 3 years ago

I see tekton-container-registry-auth is empty image

What should I do to fill that?

maiakhoa commented 3 years ago

I got this message from secret-infra

"status update failed for externalsecret jx-staging/tekton-container-registry-auth, due to modification, new poller should start"}

EDIT: do we have any way to repopulate secret?

ankitm123 commented 3 years ago

do we have any way to repopulate secret?

You can make a dummy commit in the cluster git repository (the one generated from jx3-eks-vault), and it should regenerate external secrets. Would be good to have a gist of the logs from the PR you open.

Regarding the message from secret-infra, I see this: https://kubernetes.slack.com/archives/C017BF84G2Y/p1597264180001500

I see tekton-container-registry-auth is empty

Again, not sure why that is the case, something is off with external secrets in ur cluster it seems ...

ankitm123 commented 3 years ago

FWIW,

it normally has 

{
 "credHelpers":{
  "insert-aws-account-id": "ecr-login"
} 
}

This gets eventually used by kaniko as it has ecr cred helpers inside the kaniko image: https://github.com/GoogleContainerTools/kaniko/blob/master/deploy/Dockerfile#L39-L40

maiakhoa commented 3 years ago 
{
  "credHelpers": {
    "225394301252.dkr.ecr.us-east-1.amazonaws.com": "ecr-login"
  },

  "auths":{
    "ghcr.io": {
        "auth": "Z3QtamVua2luc1gtYm90OmdocF96bUhnbURLaGhPTEdsa1o3Rk9PSm5hTWxYU0Rxa0oyNVPXTVA="
    }
  }
}

it is showing like above now. But I still got the error

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: getting tag for destination: repository can only contain the runes

Should I wait the new image Kaniko

ankitm123 commented 3 years ago

it is showing like above now

That's correct.

Should I wait the new image Kaniko

No, an upgrade for kaniko image wont fix the problem. 1.3.0 will work for your usecase.

But I still got the error

https://github.com/jx3-gitops-repositories/jx3-terraform-eks/issues/24#issuecomment-921633962

maiakhoa commented 3 years ago

could you please guide me how to print that value? I tried a few way but got the error The execution of the pipeline has stopped.

ankitm123 commented 3 years ago

Which step does your pipeline fail at? check registry or build-container-build If it's build-container-build, then replace

- name: build-container-build
              resources: {}

with 

 - image: gcr.io/kaniko-project/executor:debug-v1.3.0
          name: build-container-build
          resources: {}
          script: |
            #!/busybox/sh
            source .jx/variables.sh
            echo $PUSH_CONTAINER_REGISTRY
            echo $DOCKER_REGISTRY_ORG
            cp /tekton/creds-secrets/tekton-container-registry-auth/.dockerconfigjson /kaniko/.docker/config.json
            /kaniko/executor $KANIKO_FLAGS --context=/workspace/source --dockerfile=${DOCKERFILE_PATH:-Dockerfile} --destination=$PUSH_CONTAINER_REGISTRY/$DOCKER_REGISTRY_ORG/$APP_NAME:$VERSION

Redact any sensitive information. Source: https://github.com/jenkins-x/jx3-pipeline-catalog/blob/master/tasks/maven-java11/pullrequest.yaml#L61-L68

maiakhoa commented 3 years ago

thank @ankitm123 

225394301252.dkr.ecr.us-east-1.amazonaws.com
demo-repo
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: getting tag for destination: repository can only contain the runes `abcdefghijklmnopqrstuvwxyz0123456789_-./`

it is showing correct.

ankitm123 commented 3 years ago

can you try to manually do a push to that ecr repository (docker push) and verify it works. This is not a JX issue, but something is off with the repository name or something it seems ... Also can you print out: $APP_NAME and $VERSION

maiakhoa commented 3 years ago

yes currently I'm manually pushing the build after Jenkins die. So I'm sure it's workingfield. It is showing correct. Should we about capitalization in project name field?

225394301253.dkr.ecr.us-east-1.amazonaws.com
demo-repo
Backend-Project
1.0.81
ankitm123 commented 3 years ago

So, try all small for Backend-Project:

repository can only contain the runes `abcdefghijklmnopqrstuvwxyz0123456789_-./

According to the error message above, I dont see caps being allowed by docker registry/kaniko/ecr validation logic...

it should work I think.

maiakhoa commented 3 years ago

ah thank, let me figure out how to make it small. I just switched to Pipelines for the first time. should I change in cluster git?

ankitm123 commented 3 years ago

Is that the name of the git repository? To check if that works, you can manually set that value in your pipeline.

export APP_name="backend-project"

before kaniko pushes the image.

maiakhoa commented 3 years ago

I just thought there is an official way to do that thank you so much! It is working perfectly. It is painful working as manual Jenkins builder for my team.

ankitm123 commented 3 years ago

I just thought there is an official way to do that

Not sure what u mean by official way. JenkinsX just tries to find that value from the git repository name you have. So, if your repository is named Backend-Project, it uses that. As I mentioned, you can over write it using APP_NAME.

This is the code where it sets those values: https://github.com/jenkins-x-plugins/jx-gitops/blob/main/pkg/cmd/variables/variables.go#L198-L200 which in turn gets it's value from: https://github.com/jenkins-x/jx-helpers/blob/6eb9664b305cfe79adad52bd7ad65811347359ca/pkg/scmhelpers/discover.go#L161

Having said that, I think we could add validation around git repository names, to help the user debug the issues. Or we could just internally change the git repository name to all small here: https://github.com/jenkins-x-plugins/jx-gitops/blob/main/pkg/cmd/variables/variables.go#L200

Will open a PR tonight for this.

ankitm123 commented 3 years ago

Btw, it would be awesome if you could all this information in the faq section of the docs to help other users :)

ankitm123 commented 3 years ago

Also for the sake of completeness, could you try and see if this issue persists (without setting the environment variable for APP_NAME) in the latest kaniko version (1.6.0)?

maiakhoa commented 3 years ago

yes @ankitm123 let me do that

maiakhoa commented 3 years ago

sorry I got another issue when trying to build the new release. It's lacking hard disk of pod. I tried to increase in Cluster Git but it is resetting to 8Gi as default. So I can not try kaniko version 1.6.0 yet

Let me play around first

maiakhoa commented 3 years ago

I tried with gcr.io/kaniko-project/executor:v1.6.0-debug but still got the same error if it is not set environment

ankitm123 commented 3 years ago

ok that is good to know. I think JX can show some warning to end users or convert the repo name to all small. I will open a PR addressing this issue soon. For now, we can close this issueas it has been resolved.