Closed thavlik closed 3 years ago
dacleyra
commented
5 years ago as a guess, pipeline resource may have to have a url in the form of git@github.com/thavlik/my-git.git
https://github.com/tektoncd/pipeline/blob/master/examples/taskruns/taskrun-git-ssh.yaml#L40
thavlik
commented
5 years ago Thanks for the tip! I will give this a try and post back!
rakhbari
commented
5 years ago I have the same ErrImagePull issue as @thavlik and mine has nothing to do with Kaniko. It's a fairly straight-forward Task and TaskRun that's attempting to pull a builder image and issue commands to it to build app JAR artifact.
apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
name: tektondemo-source-git
spec:
type: git
params:
- name: revision
value: master
- name: url
value: https://github.corp.ebay.com/cna-working-group/tektoncd-poc
---
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
name: build-tektondemo-artifacts
spec:
inputs:
resources:
- name: workspace
type: git
targetPath: source
params:
- name: raptorio-builder
description: The path to the raptor-io build image
default: ecr.vip.ebayc3.com/ciaas/raptorio-builder
steps:
- name: mvn-wrapper
image: ${inputs.params.raptorio-builder}
command: ["/bin/bash"]
args: ["-c", "mvn -N io.takari:maven:wrapper -Dmaven=3.5.2"]
- name: mvn-chmod
image: ${inputs.params.raptorio-builder}
command: ["/bin/bash"]
args: ["-c", "chmod +x /workspace/source/mvnw"]
- name: mvn-build
image: ${inputs.params.raptorio-builder}
command: ['/bin/bash']
args: ["-c", "cd /workspace/source && ./mvnw clean install -s settings.xml"]
- name: list-jar
image: ${inputs.params.raptorio-builder}
command: ['/bin/bash']
args: ["-c", "cd /raptor-io-workspace && ls -l"]
volumeMounts:
- name: custom-volume
mountPath: /raptor-io-workspace
volumes:
- name: custom-volume
emptyDir: {}
---
apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
name: build-tektondemo-artifacts-task-run
spec:
serviceAccount: build-bot
taskRef:
name: build-tektondemo-artifacts
inputs:
resources:
- name: workspace
resourceRef:
name: tektondemo-source-gitHere's the build-bot account as described by kubectl:
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: "2019-08-06T20:05:26Z"
name: build-bot
namespace: default
resourceVersion: "422248"
selfLink: /api/v1/namespaces/default/serviceaccounts/build-bot
uid: 7daf64d9-378a-4395-b75d-490ff0ec8b0a
secrets:
- name: github-secret
- name: quay-secret
- name: ecr-secret
- name: build-bot-token-2m88xI have double and triple-checked the validity of all the secrets listed under secrets. They all absolutely work fine. The builder image in question is in a private internal company registry ecr.vip.ebayc3.com/ciaas/raptorio-builder. The creds that should allow it to be pulled are in ecr-secret as such:
apiVersion: v1
kind: Secret
type: kubernetes.io/basic-auth
metadata:
name: ecr-secret
annotations:
tekton.dev/docker-0: https://ecr.vip.ebayc3.com
stringData:
username: <redacted>
password: <redacted>I can successfully docker login to ecr.vip.ebayc3.com using the above credentials and do docker pull on the same image with no problems.
Yet, soon after the pod runs I get:
Normal Pulling 6m37s kubelet, tekton-cd Pulling image "ecr.vip.ebayc3.com/ciaas/raptorio-builder"
Warning Failed 6m36s kubelet, tekton-cd Error: ImagePullBackOff
Warning Failed 6m36s kubelet, tekton-cd Failed to pull image "ecr.vip.ebayc3.com/ciaas/raptorio-builder": rpc error: code = Unknown desc = failed to resolve image "ecr.vip.ebayc3.com/ciaas/raptorio-builder:latest": no available registry endpoint: failed to fetch anonymous token: unexpected status: 401 Unauthorized
Warning Failed 6m36s kubelet, tekton-cd Error: ErrImagePull
Normal BackOff 6m36s kubelet, tekton-cd Back-off pulling image "ecr.vip.ebayc3.com/ciaas/raptorio-builder"Any ideas how I can further diagnose this issue?
vdemeester
commented
5 years ago /kind bug /kind question
anxinyf
commented
5 years ago @rakhbari Hi, have you solved this problem yet? I have a same problem. When ServiceAccount like this:
apiVersion: v1
kind: ServiceAccount
metadata:
name: build-bot
secrets:
- name: docker-pass
- name: git-passbut cannot push image to destination: UNAUTHORIZED. My secret info is right.
rakhbari
commented
5 years ago @anxinyf No, unfortunately I haven't been able to get this to work. I even added imagePullSecret to my ServiceAccount definition as suggested by the developer docs:
https://github.com/tektoncd/pipeline/tree/master/docs/developers#entrypoint-rewriting-and-step-ordering
But even that didn't help. Frankly I'm not sure if this is an issue with Tekton or an issue with Quay v3. We use RedHat Quay v3 as our private Docker repo and the above developer doc clearly states that you must have an imagePullSecret defined, but like I said, it didn't help.
The only way we've been able to get past this is to make the image public in Quay. It's not a fix, just a workaround for now. I've got to put some time into deep analysis of this issue because we can't go to "production" with a requirement to make images public in Quay.
anxinyf
commented
5 years ago @rakhbari I create a configmap with file ~/.docker/config.json, then mount it to kaniko image, like this:
kubectl create configmap docker-config --from-file=/root/.docker/config.json
And,
...
env:
- name: DOCKER_CONFIG
value: "/builder/home/.docker/"
volumeMounts:
- name: docker-config
mountPath: /builder/home/.docker/
volumes:
- name: docker-config
configMap:
name: docker-configserviceAccount like:
apiVersion: v1
kind: ServiceAccount
metadata:
name: build-bot
secrets:
- name: git-passit seems to be viable for me.
but I have a question about Secret：
apiVersion: v1
kind: Secret
metadata:
annotations:
tekton.dev/git-0: https://github.com
tekton.dev/git-1: https://gitlab.com
tekton.dev/docker-0: https://gcr.io
type: kubernetes.io/basic-auth
stringData:
username: <cleartext non-encoded>
password: <cleartext non-encoded>in this Secret, git repo and docker repo have the same username and password?
vdemeester
commented
4 years ago @anxinyf indeed :) @thavlik is this still an issue ?
rakhbari
commented
4 years ago @anxinyf @vdemeester Sorry for the very late reply. This is no longer an issue for me. I stopped using PipelineResource as of pipelines v1beta1 so now I just use the image: attribute in my Task step directly and that retrieves the builder image directly from our internal Quay registry with no problems.
However, in my service account definition, I had to place my Quay secret under both imagePullSecrets and secrets. Originally I only had it just under imagePullSecrets and that didn't seem to work. I kept getting 401 errors during the TaskRun. But as soon as I added the same Quay secret under secrets, that got rid of that error and the image is able to be pulled with no problems.
tekton-robot
commented
3 years ago Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
/lifecycle stale
Send feedback to tektoncd/plumbing.
tekton-robot
commented
3 years ago Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
/lifecycle rotten
Send feedback to tektoncd/plumbing.
tekton-robot
commented
3 years ago Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.
/close
Send feedback to tektoncd/plumbing.
tekton-robot
commented
3 years ago @tekton-robot: Closing this issue.
Expected Behavior
A task should use secrets as per the documentation.
Actual Behavior
The TaskRun finishes with this error (as per
kubectl get tr build-my-image-task-run -o yaml):Output of
kubectl describe pod:Steps to Reproduce the Problem
secretssection of the ServiceAccount used in the TaskRunkubectl describeon the TaskRun-created pod:Additional Info
I am currently trying to use a kaniko image from a private registry, but I have also used the one from gcr.io and ran into git SSH issues suggesting the container is not utilizing any of the secrets. So the issue seems pertinent to secrets in general.
Here is the manifest for the SSH secret and service account:
If necessary, here is the Task and TaskRun: