search

tektoncd / chains

Supply Chain Security in Tekton Pipelines
Apache License 2.0
245 stars 126 forks source link

Add `artifacts.taskrun.enabled` and `artifacts.oci.enabled` config options to configure artifacts for signing #279

Closed mtcolman closed 2 years ago

mtcolman commented 2 years ago

Hi, I'm not sure if this is a bug or not, so raising in free-form, apologies if I've used the wrong thing.

I've followed the tutorial here: https://github.com/tektoncd/chains/blob/main/docs/tutorials/getting-started-tutorial.md

And at the end of it I get:

$ cosign verify-blob -key cosign.pub -signature ./signature ./payload 
Verified OK

However, when having a look around with:

$ oc get taskrun $TASKRUN -o=json | jq  -r ".metadata.annotations"

I can see that :

{
  "chains.tekton.dev/cert-taskrun-d72c4b52-569a-4e42-b030-0e4d5e6931fe": "",
  "chains.tekton.dev/chain-taskrun-d72c4b52-569a-4e42-b030-0e4d5e6931fe": "",
  "chains.tekton.dev/payload-taskrun-d72c4b52-569a-4e42-b030-0e4d5e6931fe": "<I've removed for brevity>",
  "chains.tekton.dev/retries": "3",
  "chains.tekton.dev/signature-taskrun-d72c4b52-569a-4e42-b030-0e4d5e6931fe": "<I've removed for brevity>",
  "chains.tekton.dev/signed": "failed",
  "pipeline.tekton.dev/release": "v0.22.0"
}

Shows:

"chains.tekton.dev/retries": "3",
chains.tekton.dev/signed": "failed"

Has something gone wrong here? And if so, why did the cosign verify-blob return Verified OK?

Thanks!

pxp928 commented 2 years ago

Based on the error messages it seems like there is an issue with the gcr registry that the example taskrun uses. If i change the registry to something else instead of gcr.io/foo/bar it changes to chains.tekton.dev/signed: "true"

    outputs:
    - name: builtImage
      resourceSpec:
        type: image
        params:
        - name: url
          value: gcr.io/foo/bar

Though it still fails the cosign verify-blob

priyawadhwa commented 2 years ago

Hey @mtcolman thanks for pointing this out! My guess is the tutorial needs to be updated and Chains is trying to upload the signature for the image to an OCI registry (which it doesn't have auth for, so it fails). But, it first stores the signature and payload on the TaskRun correctly, which is why verify-blob passes.

Could you provide the output of:

kubectl describe cm -n tekton-chains chains-config

that should confirm my suspicion!

pxp928 commented 2 years ago

Hey @priyawadhwa I was having similar issues but my cosign verify-blob also fails to verify. I am using the same taskrun given in the tutorial.

The chains controller is throwing error:

{"level":"error","ts":"2021-11-09T18:00:14.280Z","logger":"watcher","caller":"taskrun/reconciler.go:308","msg":"Returned an error","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"bd8b68a1-f7ef-4893-8ffa-b017ab731dad","knative.dev/key":"default/build-push-run-output-image-bfxtg","targetMethod":"ReconcileKind","error":"1 error occurred:\n\t* uploading: GET https://gcr.io/v2/token?scope=repository%3Afoo%2Fbar%3Apull&service=gcr.io: UNKNOWN: Service 'containerregistry.googleapis.com' is not enabled for consumer 'project:foo'.\n\n","stacktrace":"github.com/tektoncd/chains/vendor/github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun.(*reconcilerImpl).Reconcile\n\tgithub.com/tektoncd/chains/vendor/github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun/reconciler.go:308\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:542\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).RunContext.func3\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:478"}
{"level":"error","ts":"2021-11-09T18:00:14.285Z","logger":"watcher","caller":"controller/controller.go:566","msg":"Reconcile error","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","duration":0.2414034,"error":"1 error occurred:\n\t* uploading: GET https://gcr.io/v2/token?scope=repository%3Afoo%2Fbar%3Apull&service=gcr.io: UNKNOWN: Service 'containerregistry.googleapis.com' is not enabled for consumer 'project:foo'.\n\n","stacktrace":"github.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:566\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:543\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).RunContext.func3\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:478"}
{"level":"info","ts":"2021-11-09T18:00:14.285Z","logger":"watcher","caller":"taskrun/taskrun.go:57","msg":"taskrun default/build-push-run-output-image-bfxtg has been reconciled","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"4ab64e15-4382-4b93-8782-cee8678cd3e5","knative.dev/key":"default/build-push-run-output-image-bfxtg"}
{"level":"info","ts":"2021-11-09T18:00:14.285Z","logger":"watcher","caller":"controller/controller.go:550","msg":"Reconcile succeeded","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"4ab64e15-4382-4b93-8782-cee8678cd3e5","knative.dev/key":"default/build-push-run-output-image-bfxtg","duration":0.000104}
{"level":"info","ts":"2021-11-09T18:00:14.285Z","logger":"watcher.event-broadcaster","caller":"record/event.go:282","msg":"Event(v1.ObjectReference{Kind:\"TaskRun\", Namespace:\"default\", Name:\"build-push-run-output-image-bfxtg\", UID:\"6f0edb08-b19a-4106-8c3c-920e82d21a90\", APIVersion:\"tekton.dev/v1beta1\", ResourceVersion:\"373438\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' 1 error occurred:\n\t* uploading: GET https://gcr.io/v2/token?scope=repository%3Afoo%2Fbar%3Apull&service=gcr.io: UNKNOWN: Service 'containerregistry.googleapis.com' is not enabled for consumer 'project:foo'.\n\n","commit":"9bd7fb8"}

Here is my configmap for chains:

Name:         chains-config
Namespace:    tekton-chains
Labels:       app.kubernetes.io/component=chains
              app.kubernetes.io/instance=default
              app.kubernetes.io/part-of=tekton-pipelines
              pipeline.tekton.dev/release=devel
              version=v0.5.0
Annotations:  <none>

Data
====
artifacts.taskrun.format:
----
tekton-provenance
artifacts.taskrun.storage:
----
tekton
Events:  <none>

Cosgin Verify Error:

cosign verify-blob -key cosign.pub -signature ./signature ./payload
error: verifying blob [./payload]: failed to verify signature
mtcolman commented 2 years ago

Hey @mtcolman thanks for pointing this out! My guess is the tutorial needs to be updated and Chains is trying to upload the signature for the image to an OCI registry (which it doesn't have auth for, so it fails). But, it first stores the signature and payload on the TaskRun correctly, which is why verify-blob passes.

Could you provide the output of:

kubectl describe cm -n tekton-chains chains-config

that should confirm my suspicion!

Hi, @priyawadhwa I subsequently ran through the in-toto tutorial, so my configmap says:

$ kubectl describe cm -n tekton-chains chains-config

Name:         chains-config
Namespace:    tekton-chains
Labels:       app.kubernetes.io/component=chains
              app.kubernetes.io/instance=default
              app.kubernetes.io/part-of=tekton-pipelines
              pipeline.tekton.dev/release=devel
              version=v0.5.0
Annotations:  <none>

Data
====
artifacts.taskrun.storage:
----
oci
transparency.enabled:
----
true
artifacts.taskrun.format:
----
tekton-provenance
Events:  <none>

But yes, I haven't set anything up for pushing to repo in the first tutorial, so I agree with you.

pxp928 commented 2 years ago

Hey @priyawadhwa I was having similar issues but my cosign verify-blob also fails to verify. I am using the same taskrun given in the tutorial.

The chains controller is throwing error:

{"level":"error","ts":"2021-11-09T18:00:14.280Z","logger":"watcher","caller":"taskrun/reconciler.go:308","msg":"Returned an error","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"bd8b68a1-f7ef-4893-8ffa-b017ab731dad","knative.dev/key":"default/build-push-run-output-image-bfxtg","targetMethod":"ReconcileKind","error":"1 error occurred:\n\t* uploading: GET https://gcr.io/v2/token?scope=repository%3Afoo%2Fbar%3Apull&service=gcr.io: UNKNOWN: Service 'containerregistry.googleapis.com' is not enabled for consumer 'project:foo'.\n\n","stacktrace":"github.com/tektoncd/chains/vendor/github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun.(*reconcilerImpl).Reconcile\n\tgithub.com/tektoncd/chains/vendor/github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun/reconciler.go:308\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:542\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).RunContext.func3\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:478"}
{"level":"error","ts":"2021-11-09T18:00:14.285Z","logger":"watcher","caller":"controller/controller.go:566","msg":"Reconcile error","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","duration":0.2414034,"error":"1 error occurred:\n\t* uploading: GET https://gcr.io/v2/token?scope=repository%3Afoo%2Fbar%3Apull&service=gcr.io: UNKNOWN: Service 'containerregistry.googleapis.com' is not enabled for consumer 'project:foo'.\n\n","stacktrace":"github.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:566\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:543\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).RunContext.func3\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:478"}
{"level":"info","ts":"2021-11-09T18:00:14.285Z","logger":"watcher","caller":"taskrun/taskrun.go:57","msg":"taskrun default/build-push-run-output-image-bfxtg has been reconciled","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"4ab64e15-4382-4b93-8782-cee8678cd3e5","knative.dev/key":"default/build-push-run-output-image-bfxtg"}
{"level":"info","ts":"2021-11-09T18:00:14.285Z","logger":"watcher","caller":"controller/controller.go:550","msg":"Reconcile succeeded","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"4ab64e15-4382-4b93-8782-cee8678cd3e5","knative.dev/key":"default/build-push-run-output-image-bfxtg","duration":0.000104}
{"level":"info","ts":"2021-11-09T18:00:14.285Z","logger":"watcher.event-broadcaster","caller":"record/event.go:282","msg":"Event(v1.ObjectReference{Kind:\"TaskRun\", Namespace:\"default\", Name:\"build-push-run-output-image-bfxtg\", UID:\"6f0edb08-b19a-4106-8c3c-920e82d21a90\", APIVersion:\"tekton.dev/v1beta1\", ResourceVersion:\"373438\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' 1 error occurred:\n\t* uploading: GET https://gcr.io/v2/token?scope=repository%3Afoo%2Fbar%3Apull&service=gcr.io: UNKNOWN: Service 'containerregistry.googleapis.com' is not enabled for consumer 'project:foo'.\n\n","commit":"9bd7fb8"}

Here is my configmap for chains:

Name:         chains-config
Namespace:    tekton-chains
Labels:       app.kubernetes.io/component=chains
              app.kubernetes.io/instance=default
              app.kubernetes.io/part-of=tekton-pipelines
              pipeline.tekton.dev/release=devel
              version=v0.5.0
Annotations:  <none>

Data
====
artifacts.taskrun.format:
----
tekton-provenance
artifacts.taskrun.storage:
----
tekton
Events:  <none>

Cosgin Verify Error:

cosign verify-blob -key cosign.pub -signature ./signature ./payload
error: verifying blob [./payload]: failed to verify signature

Figured out my issue was that i was using an old version of cosign. Once I updated it solved the issue.

priyawadhwa commented 2 years ago

@pxp928 glad it's working again! @mtcolman yup looks like it's trying to store the signature in oci -- i think if you change the storage back to tekton it should work --

kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.taskrun.storage": "tekton"}}'

let me know if you still see errors!

mtcolman commented 2 years ago

@priyawadhwa I think I might still have this issue...

I'm running the "getting started" tutorial.

And this is the current state of tekton-chains configmap "chains-config" (i.e. storage is Tekton):

kubectl describe cm -n tekton-chains chains-config
Name:         chains-config
Namespace:    tekton-chains
Labels:       app.kubernetes.io/component=chains
              app.kubernetes.io/instance=default
              app.kubernetes.io/part-of=tekton-pipelines
              pipeline.tekton.dev/release=devel
              version=v0.5.0
Annotations:  <none>

Data
====
artifacts.taskrun.format:
----
tekton-provenance
artifacts.taskrun.storage:
----
tekton
transparency.enabled:
----
true
Events:  <none>

I've just run: kubectl create -f https://raw.githubusercontent.com/tektoncd/chains/main/examples/taskruns/task-output-image.yaml and it tells me taskrun.tekton.dev/build-push-run-output-image-s98hw created.

So I use the taskRun ID in this command:

oc get tr build-push-run-output-image-s98hw -o=json | jq -r ".metadata.annotations[\"chains.tekton.dev/signed\"]"
failed

And it informs me the status is failed. ("chains.tekton.dev/signed": "failed").

Here's the full output of annotations:

oc get tr build-push-run-output-image-s98hw -o=json | jq -r ".metadata.annotations"
{
  "chains.tekton.dev/cert-taskrun-1ba06026-0865-4c02-94cf-3e5bdf192107": "",
  "chains.tekton.dev/chain-taskrun-1ba06026-0865-4c02-94cf-3e5bdf192107": "",
  "chains.tekton.dev/payload-taskrun-1ba06026-0865-4c02-94cf-3e5bdf192107": "<removed for brevity>",
  "chains.tekton.dev/retries": "3",
  "chains.tekton.dev/signature-taskrun-1ba06026-0865-4c02-94cf-3e5bdf192107": "<removed for brevity>",
  "chains.tekton.dev/signed": "failed",
  "chains.tekton.dev/transparency": "https://rekor.sigstore.dev/858199",
  "pipeline.tekton.dev/release": "v0.22.0"
}

And here's me double checking I'm definitely looking at the taskRun I've just asked to be created:

oc get tr build-push-run-output-image-s98hw
NAME                                SUCCEEDED   REASON      STARTTIME   COMPLETIONTIME
build-push-run-output-image-s98hw   True        Succeeded   9m16s       8m51s

This is the output of oc get events:

oc get events
LAST SEEN   TYPE      REASON            OBJECT                                            MESSAGE
20m         Normal    Scheduled         pod/build-push-run-output-image-s98hw-pod-5tnzn   Successfully assigned tekton-chains/build-push-run-output-image-s98hw-pod-5tnzn to 10.112.78.35
20m         Normal    AddedInterface    pod/build-push-run-output-image-s98hw-pod-5tnzn   Add eth0 [172.30.224.163/32]
20m         Normal    Pulled            pod/build-push-run-output-image-s98hw-pod-5tnzn   Container image "registry.access.redhat.com/ubi8/ubi-minimal@sha256:fdfb0770bff33e0f97d78583efd68b546a19d0a4b0ac23eef25ef261bca3e975" already present on machine
20m         Normal    Created           pod/build-push-run-output-image-s98hw-pod-5tnzn   Created container place-scripts
20m         Normal    Started           pod/build-push-run-output-image-s98hw-pod-5tnzn   Started container place-scripts
20m         Normal    Pulled            pod/build-push-run-output-image-s98hw-pod-5tnzn   Container image "registry.redhat.io/openshift-pipelines/pipelines-entrypoint-rhel8@sha256:4989b910b5beae615a360deb31c8a4951e7b867a93374dd29d42815cf3480cc0" already present on machine
20m         Normal    Created           pod/build-push-run-output-image-s98hw-pod-5tnzn   Created container place-tools
20m         Normal    Started           pod/build-push-run-output-image-s98hw-pod-5tnzn   Started container place-tools
20m         Normal    Pulled            pod/build-push-run-output-image-s98hw-pod-5tnzn   Container image "registry.access.redhat.com/ubi8/ubi-minimal@sha256:fdfb0770bff33e0f97d78583efd68b546a19d0a4b0ac23eef25ef261bca3e975" already present on machine
20m         Normal    Created           pod/build-push-run-output-image-s98hw-pod-5tnzn   Created container step-create-dir-builtimage-ftmmv
20m         Normal    Started           pod/build-push-run-output-image-s98hw-pod-5tnzn   Started container step-create-dir-builtimage-ftmmv
20m         Normal    Pulled            pod/build-push-run-output-image-s98hw-pod-5tnzn   Container image "registry.redhat.io/openshift-pipelines/pipelines-git-init-rhel8@sha256:afc5d3f9efe26c7042635d43b8ffd09d67936e3d0b6b901dc08a33e20313d361" already present on machine
20m         Normal    Created           pod/build-push-run-output-image-s98hw-pod-5tnzn   Created container step-git-source-sourcerepo-xcsrp
20m         Normal    Started           pod/build-push-run-output-image-s98hw-pod-5tnzn   Started container step-git-source-sourcerepo-xcsrp
20m         Normal    Pulling           pod/build-push-run-output-image-s98hw-pod-5tnzn   Pulling image "busybox"
20m         Normal    Pulled            pod/build-push-run-output-image-s98hw-pod-5tnzn   Successfully pulled image "busybox" in 4.555106292s
20m         Normal    Created           pod/build-push-run-output-image-s98hw-pod-5tnzn   Created container step-build-and-push
20m         Normal    Started           pod/build-push-run-output-image-s98hw-pod-5tnzn   Started container step-build-and-push
20m         Normal    Pulling           pod/build-push-run-output-image-s98hw-pod-5tnzn   Pulling image "busybox"
20m         Normal    Pulled            pod/build-push-run-output-image-s98hw-pod-5tnzn   Successfully pulled image "busybox" in 1.714015821s
20m         Normal    Created           pod/build-push-run-output-image-s98hw-pod-5tnzn   Created container step-echo
20m         Normal    Started           pod/build-push-run-output-image-s98hw-pod-5tnzn   Started container step-echo
20m         Normal    Pulled            pod/build-push-run-output-image-s98hw-pod-5tnzn   Container image "registry.redhat.io/openshift-pipelines/pipelines-imagedigestexporter-rhel8@sha256:1c8ef68a540856de81657b47c28c632f43030a2566229b0c22c5e6ab0a5adce3" already present on machine
20m         Normal    Created           pod/build-push-run-output-image-s98hw-pod-5tnzn   Created container step-image-digest-exporter-r2s2l
20m         Normal    Started           pod/build-push-run-output-image-s98hw-pod-5tnzn   Started container step-image-digest-exporter-r2s2l
20m         Normal    Started           taskrun/build-push-run-output-image-s98hw
20m         Normal    FinalizerUpdate   taskrun/build-push-run-output-image-s98hw         Updated "build-push-run-output-image-s98hw" finalizers
20m         Normal    Pending           taskrun/build-push-run-output-image-s98hw         Pending
20m         Normal    Pending           taskrun/build-push-run-output-image-s98hw         pod status "Initialized":"False"; message: "containers with incomplete status: [place-scripts place-tools]"
20m         Normal    Pending           taskrun/build-push-run-output-image-s98hw         pod status "Initialized":"False"; message: "containers with incomplete status: [place-tools]"
20m         Normal    Pending           taskrun/build-push-run-output-image-s98hw         pod status "Ready":"False"; message: "containers with unready status: [step-create-dir-builtimage-ftmmv step-git-source-sourcerepo-xcsrp step-build-and-push step-echo step-image-digest-exporter-r2s2l]"
20m         Normal    Running           taskrun/build-push-run-output-image-s98hw         Not all Steps in the Task have finished executing
19m         Normal    Succeeded         taskrun/build-push-run-output-image-s98hw         All Steps have completed executing
19m         Warning   InternalError     taskrun/build-push-run-output-image-s98hw         1 error occurred:
            * uploading: GET https://gcr.io/v2/token?scope=repository%3Afoo%2Fbar%3Apull&service=gcr.io: UNKNOWN: Service 'containerregistry.googleapis.com' is not enabled for consumer 'project:foo'.

Note the error: 

InternalError     taskrun/build-push-run-output-image-s98hw         1 error occurred:
            * uploading: GET https://gcr.io/v2/token?scope=repository%3Afoo%2Fbar%3Apull&service=gcr.io: UNKNOWN: Service 'containerregistry.googleapis.com' is not enabled for consumer 'project:foo'.

And the taskRun configuration:

    outputs:
    - name: builtImage
      resourceSpec:
        type: image
        params:
        - name: url
          value: gcr.io/foo/bar

tekton logs of taskRun doesn't seem to flag anything untoward:

tkn tr logs build-push-run-output-image-s98hw

[git-source-sourcerepo-xcsrp] {"level":"info","ts":1636970808.014247,"caller":"git/git.go:169","msg":"Successfully cloned https://github.com/GoogleContainerTools/skaffold @ 6ed7aad5e8a36052ee5f6079fc91368e362121f7 (grafted, HEAD) in path /workspace/sourcerepo"}
[git-source-sourcerepo-xcsrp] {"level":"info","ts":1636970808.6034794,"caller":"git/git.go:207","msg":"Successfully initialized and updated submodules in path /workspace/sourcerepo"}

[build-and-push] + set -e
[build-and-push] + cat

[echo] + cat /workspace/sourcerepo/index.json
[echo] {
[echo] "schemaVersion": 2,
[echo] "manifests": [
[echo]     {
[echo]     "mediaType": "application/vnd.oci.image.index.v1+json",
[echo]     "size": 314,
[echo]     "digest": "sha256:05f95b26ed10668b7183c1e2da98610e91372fa9f510046d4ce5812addad86b5"
[echo]     }
[echo] ]
[echo] }
[echo]

[image-digest-exporter-r2s2l] {"severity":"INFO","timestamp":"2021-11-15T10:06:50.254370252Z","caller":"logging/config.go:116","message":"Successfully created the logger."}
[image-digest-exporter-r2s2l] {"severity":"INFO","timestamp":"2021-11-15T10:06:50.254978848Z","caller":"logging/config.go:117","message":"Logging level set to: info"}
[image-digest-exporter-r2s2l] {"severity":"INFO","timestamp":"2021-11-15T10:06:50.255141162Z","caller":"logging/config.go:79","message":"Fetch GitHub commit ID from kodata failed","error":"\"KO_DATA_PATH\" does not exist or is empty"}

And here is some output from oc logs tekton-chains-controller-7f94b7fc57-zgfpl:

{"level":"info","ts":"2021-11-15T10:06:50.227Z","logger":"watcher","caller":"taskrun/taskrun.go:52","msg":"taskrun tekton-chains/build-push-run-output-image-s98hw is still running","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"826be2ed-43da-4724-9da1-a8dc3e01b4b0","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw"}
{"level":"info","ts":"2021-11-15T10:06:50.227Z","logger":"watcher","caller":"controller/controller.go:550","msg":"Reconcile succeeded","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"826be2ed-43da-4724-9da1-a8dc3e01b4b0","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw","duration":0.000481135}
{"level":"info","ts":"2021-11-15T10:06:51.416Z","logger":"watcher","caller":"x509/x509.go:113","msg":"Found cosign key...","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"6e1152d6-15e4-491d-8c0a-0ecf34d678a3","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw"}
{"level":"warn","ts":"2021-11-15T10:06:52.291Z","logger":"watcher","caller":"chains/signing.go:68","msg":"error configuring kms signer with config {}: no provider found for that key reference","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"6e1152d6-15e4-491d-8c0a-0ecf34d678a3","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw"}
{"level":"info","ts":"2021-11-15T10:06:52.306Z","logger":"watcher","caller":"chains/signing.go:163","msg":"Created payload of type tekton-provenance for TaskRun tekton-chains/build-push-run-output-image-s98hw","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"6e1152d6-15e4-491d-8c0a-0ecf34d678a3","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw"}
{"level":"info","ts":"2021-11-15T10:06:52.314Z","logger":"watcher","caller":"chains/signing.go:178","msg":"Using wrapped envelope signer for tekton-provenance","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"6e1152d6-15e4-491d-8c0a-0ecf34d678a3","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw"}
{"level":"info","ts":"2021-11-15T10:06:52.314Z","logger":"watcher","caller":"chains/signing.go:182","msg":"Signing object with x509","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"6e1152d6-15e4-491d-8c0a-0ecf34d678a3","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw"}
{"level":"info","ts":"2021-11-15T10:06:52.387Z","logger":"watcher","caller":"tekton/tekton.go:58","msg":"Storing payload on TaskRun tekton-chains/build-push-run-output-image-s98hw","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"6e1152d6-15e4-491d-8c0a-0ecf34d678a3","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw"}
{"level":"info","ts":"2021-11-15T10:06:53.750Z","logger":"watcher","caller":"chains/signing.go:214","msg":"Uploaded entry to https://rekor.sigstore.dev with index 858184","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"6e1152d6-15e4-491d-8c0a-0ecf34d678a3","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw"}
{"level":"info","ts":"2021-11-15T10:06:53.753Z","logger":"watcher","caller":"chains/signing.go:163","msg":"Created payload of type simplesigning for TaskRun tekton-chains/build-push-run-output-image-s98hw","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"6e1152d6-15e4-491d-8c0a-0ecf34d678a3","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw"}
{"level":"info","ts":"2021-11-15T10:06:53.753Z","logger":"watcher","caller":"chains/signing.go:182","msg":"Signing object with x509","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"6e1152d6-15e4-491d-8c0a-0ecf34d678a3","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw"}
{"level":"info","ts":"2021-11-15T10:06:53.754Z","logger":"watcher","caller":"oci/oci.go:72","msg":"Storing payload on TaskRun tekton-chains/build-push-run-output-image-s98hw","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"6e1152d6-15e4-491d-8c0a-0ecf34d678a3","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw"}
{"level":"info","ts":"2021-11-15T10:06:53.756Z","logger":"watcher","caller":"oci/oci.go:96","msg":"Uploading gcr.io/foo/bar@sha256:05f95b26ed10668b7183c1e2da98610e91372fa9f510046d4ce5812addad86b5 signature","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"6e1152d6-15e4-491d-8c0a-0ecf34d678a3","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw"}
E1115 10:06:53.801975       1 config.go:141] while trying to canonicalize : stat .: permission denied
E1115 10:06:53.802865       1 config.go:141] while trying to canonicalize : stat .: permission denied
E1115 10:06:53.803699       1 config.go:107] while trying to canonicalize : stat .: permission denied
E1115 10:06:53.803976       1 config.go:107] while trying to canonicalize : stat .: permission denied
{"level":"error","ts":"2021-11-15T10:06:54.478Z","logger":"watcher","caller":"chains/signing.go:204","msg":"uploading: GET https://gcr.io/v2/token?scope=repository%3Afoo%2Fbar%3Apull&service=gcr.io: UNKNOWN: Service 'containerregistry.googleapis.com' is not enabled for consumer 'project:foo'.","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"6e1152d6-15e4-491d-8c0a-0ecf34d678a3","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw","stacktrace":"github.com/tektoncd/chains/pkg/chains.(*TaskRunSigner).SignTaskRun\n\tgithub.com/tektoncd/chains/pkg/chains/signing.go:204\ngithub.com/tektoncd/chains/pkg/reconciler/taskrun.(*Reconciler).FinalizeKind\n\tgithub.com/tektoncd/chains/pkg/reconciler/taskrun/taskrun.go:61\ngithub.com/tektoncd/chains/pkg/reconciler/taskrun.(*Reconciler).ReconcileKind\n\tgithub.com/tektoncd/chains/pkg/reconciler/taskrun/taskrun.go:42\ngithub.com/tektoncd/chains/vendor/github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun.(*reconcilerImpl).Reconcile\n\tgithub.com/tektoncd/chains/vendor/github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun/reconciler.go:249\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:542\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).RunContext.func3\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:478"}
{"level":"info","ts":"2021-11-15T10:06:54.870Z","logger":"watcher","caller":"chains/signing.go:214","msg":"Uploaded entry to https://rekor.sigstore.dev with index 858185","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"6e1152d6-15e4-491d-8c0a-0ecf34d678a3","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw"}
{"level":"error","ts":"2021-11-15T10:06:55.027Z","logger":"watcher","caller":"taskrun/reconciler.go:308","msg":"Returned an error","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"6e1152d6-15e4-491d-8c0a-0ecf34d678a3","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw","targetMethod":"ReconcileKind","error":"1 error occurred:\n\t* uploading: GET https://gcr.io/v2/token?scope=repository%3Afoo%2Fbar%3Apull&service=gcr.io: UNKNOWN: Service 'containerregistry.googleapis.com' is not enabled for consumer 'project:foo'.\n\n","stacktrace":"github.com/tektoncd/chains/vendor/github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun.(*reconcilerImpl).Reconcile\n\tgithub.com/tektoncd/chains/vendor/github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun/reconciler.go:308\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:542\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).RunContext.func3\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:478"}
{"level":"info","ts":"2021-11-15T10:06:55.028Z","logger":"watcher.event-broadcaster","caller":"record/event.go:282","msg":"Event(v1.ObjectReference{Kind:\"TaskRun\", Namespace:\"tekton-chains\", Name:\"build-push-run-output-image-s98hw\", UID:\"1ba06026-0865-4c02-94cf-3e5bdf192107\", APIVersion:\"tekton.dev/v1beta1\", ResourceVersion:\"311229752\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' 1 error occurred:\n\t* uploading: GET https://gcr.io/v2/token?scope=repository%3Afoo%2Fbar%3Apull&service=gcr.io: UNKNOWN: Service 'containerregistry.googleapis.com' is not enabled for consumer 'project:foo'.\n\n","commit":"9bd7fb8"}
{"level":"error","ts":"2021-11-15T10:06:55.028Z","logger":"watcher","caller":"controller/controller.go:566","msg":"Reconcile error","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","duration":3.751908598,"error":"1 error occurred:\n\t* uploading: GET https://gcr.io/v2/token?scope=repository%3Afoo%2Fbar%3Apull&service=gcr.io: UNKNOWN: Service 'containerregistry.googleapis.com' is not enabled for consumer 'project:foo'.\n\n","stacktrace":"github.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:566\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:543\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).RunContext.func3\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:478"}
{"level":"info","ts":"2021-11-15T10:06:55.174Z","logger":"watcher","caller":"x509/x509.go:113","msg":"Found cosign key...","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"1870f742-9602-41f7-a665-076e961362fb","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw"}
{"level":"warn","ts":"2021-11-15T10:06:55.479Z","logger":"watcher","caller":"chains/signing.go:68","msg":"error configuring kms signer with config {}: no provider found for that key reference","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"1870f742-9602-41f7-a665-076e961362fb","knative.dev/key":"tekton-chains/build-push-run-output-image-s98hw"}
priyawadhwa commented 2 years ago

Hey @mtcolman thanks for all the info! Chains is still trying to upload to oci even though you haven't specified that you want that in the config because it is still assuming the oci config defaults.

We probably need some way to tell Chains to ignore OCI artifacts altogether (or any type of artifact really!). Maybe adding a artifacts.taskrun.enabled and an artifact.oci.enabled field. Then in the tutorial we could set artifact.oci.enabled=false and this error shouldn't happen.

I'll go ahead and change the name of this issue to reflect the fix we'd need!

pxp928 commented 2 years ago

@priyawadhwa I can work on fixing this issue if no one else has started yet.

priyawadhwa commented 2 years ago

@pxp928 thank you! I'll assign you :)

concaf commented 2 years ago

@pxp928 now that #302 is merged, can we update the tutorials to reflect this?

pxp928 commented 2 years ago

@concaf the config document: https://github.com/tektoncd/chains/blob/main/docs/config.md and the tutorials: https://github.com/tektoncd/chains/tree/main/docs/tutorials were both updated as part of the PR. Is there another location that I missed?

concaf commented 2 years ago

@pxp928 ah! I see that the getting started tutorial was updated. I was looking for this in the signed provenance tutorial, shouldn't that be updated as well? 🤔

pxp928 commented 2 years ago

@pxp928 ah! I see that the getting started tutorial was updated. I was looking for this in the signed provenance tutorial, shouldn't that be updated as well? 🤔

@concaf That will still work for that tutorial.

priyawadhwa commented 2 years ago

@concaf @pxp928 I believe this issue can be closed now, can you confirm?

pxp928 commented 2 years ago

@priyawadhwa I agree it can be closed.

concaf commented 2 years ago

@priyawadhwa sounds good! 👍🏼