Closed mlbiam closed 2 years ago
@mlbiam thanks for the issue. Is there anymore logs ? The message is a warning and shouldn't prevent the rest of the step to execute. The rest of the step(s) might fail for a valid reason linked to it (most likely because the credentials are not found, …) but it could be worked-around I think (by mounting the secret instead of relying to the annotated secret).
@vdemeester You're right. The task runs but fails because it can't talk to the remote git repository. Here are the logs:
$ tkn pipelinerun logs build-hello-pipeline-run -n python-hello-build
task generate-image-tag has failed: "step-git-source-git-resource-5s5r8" exited with code 1 (image: "gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init@sha256:39b53980b255ef8af80086ac702f5b113d32f60cc4c8876b096ba37fc5912c81"); for logs run: kubectl -n python-hello-build logs build-hello-pipeline-run-generate-image-tag-4r659-pod-kq5dv -c step-git-source-git-resource-5s5r8
[generate-image-tag : create-dir-result-image-njhbx] 2021/09/28 18:48:27 warning: unsuccessful cred copy: ".ssh" from "/tekton/creds" to "/": unable to create destination directory: mkdir /.ssh: permission denied
[generate-image-tag : git-source-git-resource-5s5r8] {"level":"error","ts":1632854913.79133,"caller":"git/git.go:54","msg":"Error running git [fetch --recurse-submodules=yes --depth=1 origin --update-head-ok --force master]: exit status 128\nNo user exists for uid 1000\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:54\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:149\nmain.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:53\nruntime.main\n\truntime/proc.go:225"}
[generate-image-tag : git-source-git-resource-5s5r8] {"level":"fatal","ts":1632854913.7915208,"caller":"git-init/main.go:54","msg":"Error fetching git repository: failed to fetch [master]: exit status 128","stacktrace":"main.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:54\nruntime.main\n\truntime/proc.go:225"}
failed to get logs for task generate-image-tag : container step-git-source-git-resource-5s5r8 has failed : [{"key":"StartedAt","value":"2021-09-28T18:48:33.686Z","type":3}]
Tasks Completed: 1 (Failed: 1, Cancelled 0), Skipped: 2
Here's the Task
: https://github.com/PacktPublishing/Kubernetes---An-Enterprise-Guide-2E/blob/main/chapter14/example-apps/tekton/tekton-task1.yaml
Manually mounting the Secret
was my next thought. I wasn't sure if there was another way to go. I figured this is similar to how OpenShift runs (injecting random runAsUser) so maybe there was a way around this warning i was missing.
My workaround for now was to disable gatekeeper in namespaces that run pipelines. Looking at the Pod
objects being generated, it looks like they're running with suplemental group id, but no user id or group id.
I checked out the Dockerfile
for the git-init and checkout images, and they add user and group 65532, but they stay as root. I tried changing my mutation to assign 65532 as the default runAsUser
/runAsGroup
and no longer got the unsuccessful cred copy error, but the git checkout fails:
{"level":"error","ts":1633090656.5643337,"caller":"git/git.go:54","msg":"Error running git [fetch --recurse-submodules=yes --depth=1 origin --update-head-ok --force main]: exit status 128\nWarning: Permanently added 'gitlab-gitlab-shell.gitlab.svc.cluster.local,10.100.8.4' (ECDSA) to the list of known hosts.\r\ngit@gitlab-gitlab-shell.gitlab.svc.cluster.local: Permission denied (publickey,keyboard-interactive).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:54\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:149\nmain.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:53\nruntime.main\n\truntime/proc.go:225"}
{"level":"fatal","ts":1633090656.5645268,"caller":"git-init/main.go:54","msg":"Error fetching git repository: failed to fetch [main]: exit status 128","stacktrace":"main.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:54\nruntime.main\n\truntime/proc.go:225"}
My guess is that because user 65532's home dir isn't /tekton/creds
it isn't picking up the private key? (i think this would be a great feature and i have a dev environment so i'm happy to help make this work)
Thanks @mlbiam - thanks for the updates and offer to help!
As a side note, it looks like you are using PipelineResources
which we are likely to be deprecate.
I didn't really dive into this, but I wonder if the fact that we do not set the HOME
variable by default anymore might have an impact here @sbwsg
@afrittoli
As a side note, it looks like you are using PipelineResources which we are likely to be deprecate.
Can you point me to some background? If PipelineResources
is going away, what's the right approach? embedding the config directly into the PipelineRun
?
Here's the rationale for PipelineResources
being deprecated: https://github.com/tektoncd/community/blob/main/teps/0074-deprecate-pipelineresources.md#motivation
For cloning from a git repo there's a task in the catalog, git-clone, that accepts credentials directly as a workspace. You can pass in a k8s Secret here. When running as a non-root user you will likely also need to set the userHome
parameter.
@mlbiam I'm going to close this issue on the basis that pipelineresources are likely to be deprecated. Recommend using the git-clone
task from the catalog instead. Please feel free to reopen this issue if you'd like more help around this. Cheers!
/close
@sbwsg: Closing this issue.
Expected Behavior
We are using GateKeeper to enforce pod security policies in a KinD cluster (1.21.1). A mutating webhook adds
runAsUser
,runAsGroup
, andfsUser
to all containers and initContainers if one isn't present. When ourPipeline
launches we expect everything to run.Actual Behavior
our pipelines show:
Steps to Reproduce the Problem
Additional Info
Kubernetes version:
Output of
kubectl version
:Here's the complete
Pod
listing: