What did I miss??

[maorkuriel]

Question I am running the ./update-and-run.sh script and failing with the following error:

kubernetes/tekton-resources/demo on  main [!?] took 2s ❯ ./update-and-run.sh
Error: uninstall: Release not loaded: gatekeeper-template: release: not found Error: uninstall: Release not loaded: demo: release: not found Release "gatekeeper-template" does not exist. Installing it now. NAME: gatekeeper-template LAST DEPLOYED: Mon Nov 1 12:54:09 2021 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None Wait 5 seconds for gatekeeper-constraint-templates to become available Release "demo" does not exist. Installing it now. Error: rendered manifests contain a resource that already exists. Unable to continue with install: Task "kaniko" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "demo"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default"

[joshualucas84]

@maorkuriel, are you having this issue off of the main branch? because this should've been resolved.

[maorkuriel]

Yes, @joshualucas84, this is from the main branch. I cloned it two hours ago and ran the scripts.

[joshualucas84]

Roger that, @maorkuriel could you run in the same directory helm upgrade -i demo image-verification --values image-verification/values.yaml --debug and post the results?

[maorkuriel]

Here you go @joshualucas84

kubernetes/tekton-resources/demo on  main [!?] ❯ helm upgrade -i demo image-verification --values image-verification/values.yaml --debug history.go:56: [debug] getting history for release demo Release "demo" does not exist. Installing it now. install.go:178: [debug] Original chart version: "" install.go:199: [debug] CHART PATH: /Users/maorkuriel/Documents/GitHub/ssf/kubernetes/tekton-resources/demo/image-verification

Error: rendered manifests contain a resource that already exists. Unable to continue with install: Task "kaniko" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "demo"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default" helm.go:88: [debug] Task "kaniko" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "demo"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default" rendered manifests contain a resource that already exists. Unable to continue with install helm.sh/helm/v3/pkg/action.(Install).RunWithContext helm.sh/helm/v3/pkg/action/install.go:295 main.runInstall helm.sh/helm/v3/cmd/helm/install.go:265 main.newUpgradeCmd.func2 helm.sh/helm/v3/cmd/helm/upgrade.go:124 github.com/spf13/cobra.(Command).execute github.com/spf13/cobra@v1.2.1/command.go:856 github.com/spf13/cobra.(Command).ExecuteC github.com/spf13/cobra@v1.2.1/command.go:974 github.com/spf13/cobra.(Command).Execute github.com/spf13/cobra@v1.2.1/command.go:902 main.main helm.sh/helm/v3/cmd/helm/helm.go:87 runtime.main runtime/proc.go:255 runtime.goexit runtime/asm_arm64.s:1133

kubernetes/tekton-resources/demo on  main [!?] took 2s ❯

[joshualucas84]

@maorkuriel , could you try the changes in this PR to see if it addresses your issues, https://github.com/thesecuresoftwarefactory/ssf/pull/28

[maorkuriel]

Hi @joshualucas84 , it didn't work, here is the output:

kubernetes/tekton-resources/demo on  main [!?] took 11s ❯ ./update-and-run.sh release "gatekeeper-template" uninstalled Error: uninstall: Release not loaded: demo: release: not found Release "gatekeeper-template" does not exist. Installing it now. NAME: gatekeeper-template LAST DEPLOYED: Mon Nov 1 15:19:29 2021 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None Wait 5 seconds for gatekeeper-constraint-templates to become available Release "demo" does not exist. Installing it now. Error: rendered manifests contain a resource that already exists. Unable to continue with install: Task "kaniko" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "demo"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default"

kubernetes/tekton-resources/demo on  main [!?] took 11s ❯ helm upgrade -i demo image-verification --values image-verification/values.yaml --debug history.go:56: [debug] getting history for release demo Release "demo" does not exist. Installing it now. install.go:178: [debug] Original chart version: "" install.go:199: [debug] CHART PATH: /Users/maorkuriel/Documents/GitHub/ssf/kubernetes/tekton-resources/demo/image-verification

Error: rendered manifests contain a resource that already exists. Unable to continue with install: Task "kaniko" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "demo"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default" helm.go:88: [debug] Task "kaniko" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "demo"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default" rendered manifests contain a resource that already exists. Unable to continue with install helm.sh/helm/v3/pkg/action.(Install).RunWithContext helm.sh/helm/v3/pkg/action/install.go:295 main.runInstall helm.sh/helm/v3/cmd/helm/install.go:265 main.newUpgradeCmd.func2 helm.sh/helm/v3/cmd/helm/upgrade.go:124 github.com/spf13/cobra.(Command).execute github.com/spf13/cobra@v1.2.1/command.go:856 github.com/spf13/cobra.(Command).ExecuteC github.com/spf13/cobra@v1.2.1/command.go:974 github.com/spf13/cobra.(Command).Execute github.com/spf13/cobra@v1.2.1/command.go:902 main.main helm.sh/helm/v3/cmd/helm/helm.go:87 runtime.main runtime/proc.go:255 runtime.goexit runtime/asm_arm64.s:1133

kubernetes/tekton-resources/demo on  main [!?] took 2s ❯

[joshualucas84]

@maorkuriel , Sorry for all the trouble I was running into this issue with the pipelinerun.yaml and pipeline.yaml, ive never seen it with the kaniko task yaml. Could you run the following command for me and upload the result helm version and helm template demo image-verification --values image-verification/values.yaml ?

[maorkuriel]

Hi @joshualucas84 No problem, here is the output:

kubernetes/tekton-resources/demo on  main [!?] ❯ helm template demo image-verification --values image-verification/values.yaml

Source: image-verification/templates/pipeline.yaml

apiVersion: v1 kind: ServiceAccount metadata: name: build-bot labels: meta.helm.sh/release-name: demo meta.helm.sh/release-namespace: default app.kubernetes.io/instance: demo app.kubernetes.io/managed-by: Helm imagePullSecrets:

• name: regcred

  Below is required because Kaniko task currently doesn't use imagePullSecrets normally

  secrets:

• name: regcred

  Source: image-verification/templates/gatekeeper-signing-checker-resources.yaml

  apiVersion: v1 kind: ConfigMap metadata: name: key-map namespace: gatekeeper data: horismos_config: |- base_key_path: "/tmp/keys" key_map: image: "cosign.pub" 'sbom.sources': "cosign.sbom.pub" port: 8085 image: |- -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEoqCioewUvqHTFxPRzDdml6nCdPkDbBgbQvR1U/IBCBMrYLVXoRMTMKjN011Ce1jqj7Tj7zHtg7WXYa0bLbu8LA== -----END PUBLIC KEY----- 'sbom.sources': |- -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEoqCioewUvqHTFxPRzDdml6nCdPkDbBgbQvR1U/IBCBMrYLVXoRMTMKjN011Ce1jqj7Tj7zHtg7WXYa0bLbu8LA== -----END PUBLIC KEY-----

  Source: image-verification/templates/pipeline.yaml

  apiVersion: v1 kind: PersistentVolumeClaim metadata: name: kaniko-source-pvc spec: accessModes:

  • ReadWriteOnce resources: requests: storage: 2000Mi

    Source: image-verification/templates/gatekeeper-signing-checker-resources.yaml

    apiVersion: v1 kind: Service metadata: name: gatekeeper-signing-checker-service namespace: gatekeeper spec: selector: app: gatekeeper-signing-checker ports:

  • protocol: TCP port: 8085 targetPort: 8085

    Source: image-verification/templates/gatekeeper-signing-checker.yaml

    apiVersion: v1 kind: Service metadata: labels: name: gatekeeper-signing-checker name: gatekeeper-signing-checker namespace: gatekeeper spec: internalTrafficPolicy: Cluster ipFamilies:

• IPv4 ipFamilyPolicy: SingleStack ports:

• port: 8085 protocol: TCP targetPort: 8085 selector: run: gatekeeper-signing-checker sessionAffinity: None type: ClusterIP

  Source: image-verification/templates/gatekeeper-signing-checker.yaml

  apiVersion: apps/v1 kind: Deployment metadata: name: gatekeeper-signing-checker namespace: gatekeeper labels: name: gatekeeper-signing-checker spec: selector: matchLabels: run: gatekeeper-signing-checker replicas: 1 template: metadata: labels: run: gatekeeper-signing-checker spec: imagePullSecrets:

  • name: regcred containers:
  • name: gatekeeper-signing-checker image: ttl.sh/horismos:10m resources: limits: memory: "100Mi" cpu: "100m" ports:
    • containerPort: 8085 volumeMounts:
      • name: keys mountPath: "/tmp/keys" readOnly: true volumes:
      • name: keys configMap: name: key-map items:
      • key: "horismos_config" path: "horismos.yaml"
      • key: "image" path: "cosign.pub"

      • key: "sbom.sources" path: "cosign.sbom.pub"

        Source: image-verification/templates/gatekeeper-mutating.yaml

        apiVersion: mutations.gatekeeper.sh/v1alpha1 kind: AssignMetadata metadata: name: annotation-salsa spec: match: scope: Namespaced kinds:

      • apiGroups: ["*"] kinds: ["Pod"] namespaces:

  • "prod" location: "metadata.annotations.salsa" parameters: assign: value: "verified"

    Source: image-verification/templates/admission-control-verify-image.yaml

    apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: verify-image annotations: policies.kyverno.io/title: Verify Image policies.kyverno.io/category: Sample policies.kyverno.io/severity: medium policies.kyverno.io/subject: Pod policies.kyverno.io/minversion: 1.4.2 policies.kyverno.io/description: >- Using the Cosign project, OCI images may be signed to ensure supply chain security is maintained. Those signatures can be verified before pulling into a cluster. This policy checks the signature of an image repo called ghcr.io/kyverno/test-verify-image to ensure it has been signed by verifying its signature against the provided public key. This policy serves as an illustration for how to configure a similar rule and will require replacing with your image(s) and keys. spec: validationFailureAction: enforce background: false rules:

    • name: verify-image match: resources: kinds:
      • Pod namespaces:
      • "default" verifyImages:
  • image: "gcr.io/tekton-releases/github.com/tektoncd/*" key: |- -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEnLNw3RYx9xQjXbUEw8vonX3U4+tB kPnJq+zt386SCoG0ewIH5MB8+GjIDGArUULSDfjfM31Eae/71kavAUI0OA== -----END PUBLIC KEY-----
  • image: "gcr.io/projectsigstore/*" key: |- -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEhyQCx0E9wQWSFI9ULGwy3BuRklnt IqozONbbdbqz11hlRJy9c7SG+hdcFl9jE9uE/dwtuwU2MqU9T/cN0YkWww== -----END PUBLIC KEY-----

    Change below to your public keys if you built the images yourself.

  • image: "ghcr.io/*" key: |- -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEoqCioewUvqHTFxPRzDdml6nCdPkDbBgbQvR1U/IBCBMrYLVXoRMTMKjN011Ce1jqj7Tj7zHtg7WXYa0bLbu8LA== -----END PUBLIC KEY-----

  • image: "ttl.sh/*" key: |- -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEoqCioewUvqHTFxPRzDdml6nCdPkDbBgbQvR1U/IBCBMrYLVXoRMTMKjN011Ce1jqj7Tj7zHtg7WXYa0bLbu8LA== -----END PUBLIC KEY-----

    Source: image-verification/templates/gatekeeper-constraints.yaml

    apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sRequiredAttestations metadata: name: image-must-have-valid-slsa-attestations labels: meta.helm.sh/release-name: demo meta.helm.sh/release-namespace: default app.kubernetes.io/instance: demo app.kubernetes.io/managed-by: Helm spec: enforcementAction: deny match: kinds:

  • apiGroups: [""] kinds: ["Pod"] namespaces:

  • "prod"

    Source: image-verification/templates/gatekeeper-constraints.yaml

    apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sRequiredSignatures metadata: name: image-must-have-signature labels: meta.helm.sh/release-name: demo meta.helm.sh/release-namespace: default app.kubernetes.io/instance: demo app.kubernetes.io/managed-by: Helm spec: enforcementAction: deny match: kinds:

  • apiGroups: [""] kinds: ["Pod"] namespaces:

  • "prod" parameters: signatures: ["image"]

    Source: image-verification/templates/pipeline.yaml

    apiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: name: kaniko-cargo-pipeline spec: workspaces:

• name: shared-workspace
• name: docker-secrets
• name: sboms
• name: sbom-key params:
• name: image description: reference of the image to build tasks:
• name: fetch-repository taskRef: name: git-clone workspaces:
  • name: output workspace: shared-workspace params:
  • name: url value: https://github.com/mlieberman85/rust-hello-world-demo
  • name: subdirectory value: ""
  • name: deleteExisting value: "true"

    Below is required because upstream catalog version uses older image that isn't signed by tekton's private key

  • name: gitInitImage value: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init:v0.27.3
• name: kaniko taskRef: name: kaniko runAfter:
  • fetch-repository workspaces:
  • name: source workspace: shared-workspace
  • name: dockerconfig workspace: docker-secrets
  • name: sboms workspace: sboms
  • name: sbom-key workspace: sbom-key params:
  • name: IMAGE value: $(params.image)
  • name: EXTRA_ARGS value:
    • "--skip-tls-verify"
• name: verify-digest runAfter:
  • kaniko params:
  • name: digest value: $(tasks.kaniko.results.IMAGE_DIGEST) taskSpec: params:
    • name: digest steps:

    • name: bash image: ubuntu script: | echo $(params.digest) case .$(params.digest) in ".sha") exit 0 ;; ) echo "Digest value is not correct" && exit 1 ;; esac

      Source: image-verification/templates/pipeline-run.yaml

      apiVersion: tekton.dev/v1beta1 kind: PipelineRun metadata: name: kaniko-cargo-pipeline-run-1 labels: meta.helm.sh/release-name: demo meta.helm.sh/release-namespace: default app.kubernetes.io/instance: demo app.kubernetes.io/managed-by: Helm spec: serviceAccountName: build-bot pipelineRef: name: kaniko-cargo-pipeline params:

• name: image

  NOTE: Change me to something like below if you want to test out without access to push mlieberman85/kaniko-chains-demo

  value: ttl.sh/foo-kaniko-chains-demo:1h

  value: ttl.sh/foo-kaniko-chains-demo:1h

  workspaces:

• name: shared-workspace persistentvolumeclaim: claimName: kaniko-source-pvc
• name: docker-secrets secret: secretName: secret-dockerconfigjson
• name: sboms emptyDir: {}

• name: sbom-key secret: secretName: sbom

  Source: image-verification/templates/kaniko-with-cosign.yaml

  apiVersion: tekton.dev/v1beta1 kind: Task metadata: name: kaniko labels: app.kubernetes.io/version: "0.5" meta.helm.sh/release-name: demo meta.helm.sh/release-namespace: default app.kubernetes.io/instance: demo app.kubernetes.io/managed-by: Helm
  annotations: tekton.dev/pipelines.minVersion: "0.17.0" tekton.dev/categories: Image Build tekton.dev/displayName: "Build and upload container image using Kaniko" tekton.dev/platforms: "linux/amd64" spec: description: >- This Task builds source into a container image using Google's kaniko tool.

  Kaniko doesn't depend on a Docker daemon and executes each command within a Dockerfile completely in userspace. This enables building container images in environments that can't easily or securely run a Docker daemon, such as a standard Kubernetes cluster.

  params:

• name: IMAGE description: Name (reference) of the image to build.
• name: DOCKERFILE description: Path to the Dockerfile to build. default: ./Dockerfile
• name: CONTEXT description: The build context used by Kaniko. default: ./
• name: EXTRA_ARGS type: array default: []
• name: BUILDER_IMAGE description: The image on which builds will run (default is v1.5.1) default: gcr.io/kaniko-project/executor:v1.5.1@sha256:c6166717f7fe0b7da44908c986137ecfeab21f31ec3992f6e128fff8a94be8a5 workspaces:
• name: source description: Holds the context and docker file
• name: sboms description: Output workspace for SBOM storage
• name: sbom-key description: Signing key for sboms
• name: dockerconfig description: Includes a docker config.json optional: true mountPath: /kaniko/.docker results:
• name: IMAGE_DIGEST description: Digest of the image just built.

• name: IMAGE_URL description: URL of the image just built.

  steps:

  UNCOMMENT BELOW STEP TO TEST OUT SIGNED IMAGES/COMMENT OUT BELOW STEP TO TEST WITHOUT SIGNED IMAGES

• name: validate-parent-image-is-signed workingDir: $(workspaces.source.path)

  Replace below with your own image based on cosign docker file.

  You can do this by cloning: https://github.com/sigstore/cosign/

  And then in the cosign directory:

  docker build . --tag ttl.sh/foo-cosign:1h -f Dockerfile

  cosign sign -key k8s://tekton-chains/signing-secrets docker push ttl.sh/foo-cosign:1h

  docker push ttl.sh/foo-cosign:1h

  You can replace the tag above with whatever you want and have access to push to.

  If you are using the admission controller you need to make sure the above image is signed by cosign

  image: gcr.io/projectsigstore/cosign:v1.2.1@sha256:68801416e6ae0a48820baa3f071146d18846d8cd26ca8ec3a1e87fca8a735498 args:

  • dockerfile
  • verify
  • -key=https://raw.githubusercontent.com/mlieberman85/public-key-examples/main/cosign.image.pub
  • $(params.DOCKERFILE)
• name: create-source-sbom image: cyclonedx/cyclonedx-cli workingDir: $(workspaces.source.path) args:
  • add
  • files
  • --no-input
  • --output-format=json
  • --exclude=/.git/**
  • --output-file=$(workspaces.sboms.path)/bom-sources.json
• name: build-and-push workingDir: $(workspaces.source.path) image: $(params.BUILDER_IMAGE) args:
  • $(params.EXTRA_ARGS[*])
  • --dockerfile=$(params.DOCKERFILE)
  • --context=$(workspaces.source.path)/$(params.CONTEXT) # The user does not need to care the workspace and the source.
  • --destination=$(params.IMAGE)
  • --digest-file=/tekton/results/IMAGE_DIGEST

    kaniko assumes it is running as root, which means this example fails on platforms

    that default to run containers as random uid (like OpenShift). Adding this securityContext

    makes it explicit that it needs to run as root.

• name: write-url image: bash script: | set -e echo $(params.IMAGE) | tee $(results.IMAGE_URL.path)

  NOTE: Below is UNSAFE. It is purely used now for demoing purposes.

  A better route is to use

• name: attach-and-sign-sbom workingDir: $(workspaces.source.path) image: gcr.io/projectsigstore/cosign:v1.2.1@sha256:68801416e6ae0a48820baa3f071146d18846d8cd26ca8ec3a1e87fca8a735498 script: |

  !/busybox/sh

  set -e IMAGE_DIGEST=cat $(results.IMAGE_DIGEST.path) | sed s/:/-/ echo "Uploading: cosign upload blob -f $(workspaces.sboms.path)/bom-sources.json $(params.IMAGE):$IMAGE_DIGEST.sbom.sources" cosign upload blob -f $(workspaces.sboms.path)/bom-sources.json $(params.IMAGE):$IMAGE_DIGEST.sbom.sources echo "Signing: cosign sign blob -key $(workspaces.sbom-key.path)/cosign.key $(params.IMAGE):$IMAGE_DIGEST.sbom.sources" cosign sign -key $(workspaces.sbom-key.path)/cosign.key $(params.IMAGE):$IMAGE_DIGEST.sbom.sources

kubernetes/tekton-resources/demo on  main [!?] ❯ helm version
version.BuildInfo{Version:"v3.7.1", GitCommit:"1d11fcb5d3f3bf00dbe6fe31b8412839a96b3dc4", GitTreeState:"clean", GoVersion:"go1.17.2"}

kubernetes/tekton-resources/demo on  main [!?] ❯

[joshualucas84]

@maorkuriel run the following command kubectl delete -f https://raw.githubusercontent.com/tektoncd/catalog/v1beta1/kaniko/kaniko.yaml and then the ./update-and-run.sh and that should fix your issues for the install. Also https://github.com/thesecuresoftwarefactory/ssf/pull/19/files addresses this issue.

[maorkuriel]

Thanks @joshualucas84 , your last comment fixed the issue.

I am now faced with a new one, the pipeline fails to run and reports the following: CouldntCreateAffinityAssistantStatefulSet Failed to create StatefulSet for PipelineRun default/kaniko-cargo-pipeline-run-1 correctly: failed to create StatefulSet affinity-assistant-13a1e351e2: Internal error occurred: failed calling webhook "mutate.kyverno.svc-fail": failed to call webhook: Post "https://kyverno-svc.kyverno.svc:443/mutate?timeout=10s": EOF

If I follow the original demo flow this should not happen, can you help me with this issue?

Thanks for all your help, I appreciate your time.

[bradbeck]

@maorkuriel There was an issue in kyverno that was recently fixed. Try replacing the following lines in 30-kyverno-setup.sh:

# Assumes helm already installed by previous scripts
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm upgrade --install kyverno kyverno/kyverno -n kyverno --create-namespace --set extraArgs="{--webhooktimeout=15,--imagePullSecrets=regcred}"

with

kubectl apply -f https://raw.githubusercontent.com/kyverno/kyverno/main/config/release/install.yaml

[maorkuriel]

thank you @bradbeck , looks like this solved the issue. I will keep working with the demo env. One last question, now when I run the script I see the kaniko-cargo-pipeline-run-1 pipeline pending on the 2nd step right after it completed the fetch-reposetory step.

Looks like another issue ... when I look at the TaskRun tab I see pod status "Initialized":"False"; message: "containers with incomplete status: [place-tools place-scripts working-dir-initializer]"

[maorkuriel]

closing this ticket as a new issue was opened which address this https://github.com/thesecuresoftwarefactory/ssf/issues/32