image-provenance build failure in final step with exit code 27

[vkuznet]

Describe the bug image-provenance build failure in final step without any specific details

To Reproduce Please visit

1. Go to 'https://github.com/CHESSComputing/MetaData/actions'
2. Click on '.Actions'
3. See build failure
4. See error

Run set -euo pipefail
  set -euo pipefail
  echo "outcome=$([ "$SUCCESS" == "true" ] && echo "success" || echo "failure")" >> "$GITHUB_OUTPUT"
  [ "$CONTINUE" == "true" ] || [ "$SUCCESS" == "true" ] || exit [2](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241209868#step:2:2)7
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    BUILDER_BINARY: slsa-generator-container-linux-amd6[4](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241209868#step:2:4)
    BUILDER_DIR: internal/builders/container
    SUCCESS: false
    CONTINUE: false
Error: Process completed with exit code 2[7](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241209868#step:2:7).

Expected behavior If particular build step fails I would expect to find out details of its failure. From provided details I have no idea about what's when wrong and how to identify the issue.

Screenshots

Additional context I have suspicious that the actual issue with some github permissions as I used the same workflow in my personal repository and it is working fine. In this case, I am member of organization and need to understand which permission should be appropriate to used in this particular build step.

[vkuznet]

Seems like it is similar to the following issue https://github.com/slsa-framework/slsa-github-generator/issues/3031

[vkuznet]

Turns out error is more complicated and misleading. Upon further inspection I found that the actual error happens in image-provenance/generator step (even though it was marked as green in github web UI). In particular at this step:

Create and sign provenance
Run set -euo pipefail
set -euo pipefail

  # Generate a predicate only.
  predicate_name="predicate.json"
  "$GITHUB_WORKSPACE/$BUILDER_BINARY" generate --predicate="$predicate_name"

  COSIGN_EXPERIMENTAL=1 cosign attest --predicate="$predicate_name" \
    --type slsaprovenance \
    --yes \
    "${UNTRUSTED_IMAGE}@${UNTRUSTED_DIGEST}"
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    BUILDER_BINARY: slsa-generator-container-linux-amd64
    BUILDER_DIR: internal/builders/container
    UNTRUSTED_IMAGE: 
    UNTRUSTED_DIGEST: 
    GITHUB_CONTEXT: {
    "token": "***",
    "job": "generator",
    "ref": "refs/tags/v0.0.0-pre2",
    "sha": "4090d2b6e2c92e5056fa[3](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241207367#step:6:3)fa0ffb73a80251ba25b",
    "repository": "CHESSComputing/MetaData",
    "repository_owner": "CHESSComputing",
    "repository_owner_id": "12[4](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241207367#step:6:4)308397",
    "repositoryUrl": "git://github.com/CHESSComputing/MetaData.git",
    "run_id": "7448232909",
    "run_number": "11",
    "retention_days": "90",
    "run_attempt": "1",
    "artifact_cache_size_limit": "10",
    "repository_visibility": "public",
    "repo-self-hosted-runners-disabled": false,
    "enterprise-managed-business-id": "",
    "repository_id": "731269300",
    "actor_id": "12[5](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241207367#step:6:5)138",
    "actor": "vkuznet",
    "triggering_actor": "vkuznet",
    "workflow": "goreleaser",
    "head_ref": "",
    "base_ref": "",
    "event_name": "push",
    "event": {
      "after": "4090d2b[6](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241207367#step:6:6)e2c92e5056fa3fa0ffb[7](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241207367#step:6:7)3a[8](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241207367#step:6:8)0251ba25b",
      "base_ref": "refs/heads/main",
      "before": "0000000000000000000000000000000000000000",
      "commits": [],
      "compare": "https://github.com/CHESSComputing/MetaData/compare/v0.0.0-pre2",
      "created": true,
      "deleted": false,
      "forced": false,
      "head_commit": {
        "author": {
          "email": "xyz@abc.com",
          "name": "First Last Name",
          "username": "vkuznet"
        },
        "committer": {
          "email": "xyz@abc.com",
          "name": "First Last Name",
          "username": "vkuznet"
        },
        "distinct": true,
        "id": "40[9](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241207367#step:6:9)0d2b6e2c92e5056fa3fa0ffb73a80251ba25b",
        "message": "Add contents: write permission",
        "timestamp": "2024-01-08T08:38:14-05:00",
        "tree_id": "696aba0ffc9c458c6[11](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241207367#step:6:11)28856e8367e9bd0499402",
        "url": "https://github.com/CHESSComputing/MetaData/commit/4090d2b6e2c92e5056fa3fa0ffb73a80251ba25b"
      },
      "organization": {
        "avatar_url": "https://avatars.githubusercontent.com/u/[12](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241207367#step:6:12)4308397?v=4",
        "description": "Workflow and data processing software to reduce and analyse synchrotron experiment data",
        "events_url": "https://api.github.com/orgs/CHESSComputing/events",
        "hooks_url": "https://api.github.com/orgs/CHESSComputing/hooks",
        "id": 124308397,
        "issues_url": "https://api.github.com/orgs/CHESSComputing/issues",
        "login": "CHESSComputing",
        "members_url": "https://api.github.com/orgs/CHESSComputing/members{/member}",
        "node_id": "O_kgDOB2jLrQ",
        "public_members_url": "https://api.github.com/orgs/CHESSComputing/public_members{/member}",
        "repos_url": "https://api.github.com/orgs/CHESSComputing/repos",
        "url": "https://api.github.com/orgs/CHESSComputing"
      },
      "pusher": {
        "email": "vkuznet@users.noreply.github.com",
        "name": "vkuznet"
      },
      "ref": "refs/tags/v0.0.0-pre2",
      "repository": {
        "allow_forking": true,
        "archive_url": "https://api.github.com/repos/CHESSComputing/MetaData/{archive_format}{/ref}",
        "archived": false,
        "assignees_url": "https://api.github.com/repos/CHESSComputing/MetaData/assignees{/user}",
        "blobs_url": "https://api.github.com/repos/CHESSComputing/MetaData/git/blobs{/sha}",
        "branches_url": "https://api.github.com/repos/CHESSComputing/MetaData/branches{/branch}",
        "clone_url": "https://github.com/CHESSComputing/MetaData.git",
        "collaborators_url": "https://api.github.com/repos/CHESSComputing/MetaData/collaborators{/collaborator}",
        "comments_url": "https://api.github.com/repos/CHESSComputing/MetaData/comments{/number}",
        "commits_url": "https://api.github.com/repos/CHESSComputing/MetaData/commits{/sha}",
        "compare_url": "https://api.github.com/repos/CHESSComputing/MetaData/compare/{base}...{head}",
        "contents_url": "https://api.github.com/repos/CHESSComputing/MetaData/contents/{+path}",
        "contributors_url": "https://api.github.com/repos/CHESSComputing/MetaData/contributors",
        "created_at": 1702489643,
        "custom_properties": {},
        "default_branch": "main",
        "deployments_url": "https://api.github.com/repos/CHESSComputing/MetaData/deployments",
        "description": "MetaData service",
        "disabled": false,
        "downloads_url": "https://api.github.com/repos/CHESSComputing/MetaData/downloads",
        "events_url": "https://api.github.com/repos/CHESSComputing/MetaData/events",
        "fork": false,
        "forks": 0,
        "forks_count": 0,
        "forks_url": "https://api.github.com/repos/CHESSComputing/MetaData/forks",
        "full_name": "CHESSComputing/MetaData",
        "git_commits_url": "https://api.github.com/repos/CHESSComputing/MetaData/git/commits{/sha}",
        "git_refs_url": "https://api.github.com/repos/CHESSComputing/MetaData/git/refs{/sha}",
        "git_tags_url": "https://api.github.com/repos/CHESSComputing/MetaData/git/tags{/sha}",
        "git_url": "git://github.com/CHESSComputing/MetaData.git",
        "has_discussions": false,
        "has_downloads": true,
        "has_issues": true,
        "has_pages": false,
        "has_projects": true,
        "has_wiki": true,
        "homepage": null,
        "hooks_url": "https://api.github.com/repos/CHESSComputing/MetaData/hooks",
        "html_url": "https://github.com/CHESSComputing/MetaData",
        "id": 731269300,
        "is_template": false,
        "issue_comment_url": "https://api.github.com/repos/CHESSComputing/MetaData/issues/comments{/number}",
        "issue_events_url": "https://api.github.com/repos/CHESSComputing/MetaData/issues/events{/number}",
        "issues_url": "https://api.github.com/repos/CHESSComputing/MetaData/issues{/number}",
        "keys_url": "https://api.github.com/repos/CHESSComputing/MetaData/keys{/key_id}",
        "labels_url": "https://api.github.com/repos/CHESSComputing/MetaData/labels{/name}",
        "language": "Go",
        "languages_url": "https://api.github.com/repos/CHESSComputing/MetaData/languages",
        "license": {
          "key": "mit",
          "name": "MIT License",
          "node_id": "MDc6TGljZW5zZTEz",
          "spdx_id": "MIT",
          "url": "https://api.github.com/licenses/mit"
        },
        "master_branch": "main",
        "merges_url": "https://api.github.com/repos/CHESSComputing/MetaData/merges",
        "milestones_url": "https://api.github.com/repos/CHESSComputing/MetaData/milestones{/number}",
        "mirror_url": null,
        "name": "MetaData",
        "node_id": "R_kgDOK5ZItA",
        "notifications_url": "https://api.github.com/repos/CHESSComputing/MetaData/notifications{?since,all,participating}",
        "open_issues": 0,
        "open_issues_count": 0,
        "organization": "CHESSComputing",
        "owner": {
          "avatar_url": "https://avatars.githubusercontent.com/u/124308397?v=4",
          "email": null,
          "events_url": "https://api.github.com/users/CHESSComputing/events{/privacy}",
          "followers_url": "https://api.github.com/users/CHESSComputing/followers",
          "following_url": "https://api.github.com/users/CHESSComputing/following{/other_user}",
          "gists_url": "https://api.github.com/users/CHESSComputing/gists{/gist_id}",
          "gravatar_id": "",
          "html_url": "https://github.com/CHESSComputing",
          "id": 124308397,
          "login": "CHESSComputing",
          "name": "CHESSComputing",
          "node_id": "O_kgDOB2jLrQ",
          "organizations_url": "https://api.github.com/users/CHESSComputing/orgs",
          "received_events_url": "https://api.github.com/users/CHESSComputing/received_events",
          "repos_url": "https://api.github.com/users/CHESSComputing/repos",
          "site_admin": false,
          "starred_url": "https://api.github.com/users/CHESSComputing/starred{/owner}{/repo}",
          "subscriptions_url": "https://api.github.com/users/CHESSComputing/subscriptions",
          "type": "Organization",
          "url": "https://api.github.com/users/CHESSComputing"
        },
        "private": false,
        "pulls_url": "https://api.github.com/repos/CHESSComputing/MetaData/pulls{/number}",
        "pushed_at": 1704721128,
        "releases_url": "https://api.github.com/repos/CHESSComputing/MetaData/releases{/id}",
        "size": 588,
        "ssh_url": "git@github.com:CHESSComputing/MetaData.git",
        "stargazers": 0,
        "stargazers_count": 0,
        "stargazers_url": "https://api.github.com/repos/CHESSComputing/MetaData/stargazers",
        "statuses_url": "https://api.github.com/repos/CHESSComputing/MetaData/statuses/{sha}",
        "subscribers_url": "https://api.github.com/repos/CHESSComputing/MetaData/subscribers",
        "subscription_url": "https://api.github.com/repos/CHESSComputing/MetaData/subscription",
        "svn_url": "https://github.com/CHESSComputing/MetaData",
        "tags_url": "https://api.github.com/repos/CHESSComputing/MetaData/tags",
        "teams_url": "https://api.github.com/repos/CHESSComputing/MetaData/teams",
        "topics": [],
        "trees_url": "https://api.github.com/repos/CHESSComputing/MetaData/git/trees{/sha}",
        "updated_at": "2023-12-[13](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241207367#step:6:13)T[20](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241207367#step:6:20):01:57Z",
        "url": "https://github.com/CHESSComputing/MetaData",
        "visibility": "public",
        "watchers": 0,
        "watchers_count": 0,
        "web_commit_signoff_required": false
      },
      "sender": {
        "avatar_url": "https://avatars.githubusercontent.com/u/125138?v=4",
        "events_url": "https://api.github.com/users/vkuznet/events{/privacy}",
        "followers_url": "https://api.github.com/users/vkuznet/followers",
        "following_url": "https://api.github.com/users/vkuznet/following{/other_user}",
        "gists_url": "https://api.github.com/users/vkuznet/gists{/gist_id}",
        "gravatar_id": "",
        "html_url": "https://github.com/vkuznet",
        "id": 125138,
        "login": "vkuznet",
        "node_id": "MDQ6VXNlcjEyNTEzOA==",
        "organizations_url": "https://api.github.com/users/vkuznet/orgs",
        "received_events_url": "https://api.github.com/users/vkuznet/received_events",
        "repos_url": "https://api.github.com/users/vkuznet/repos",
        "site_admin": false,
        "starred_url": "https://api.github.com/users/vkuznet/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/vkuznet/subscriptions",
        "type": "User",
        "url": "https://api.github.com/users/vkuznet"
      }
    },
    "server_url": "https://github.com",
    "api_url": "https://api.github.com",
    "graphql_url": "https://api.github.com/graphql",
    "ref_name": "v0.0.0-pre2",
    "ref_protected": false,
    "ref_type": "tag",
    "secret_source": "Actions",
    "workflow_ref": "CHESSComputing/MetaData/.github/workflows/goreleaser.yml@refs/tags/v0.0.0-pre2",
    "workflow_sha": "4090d2b6e2c92e5056fa3fa0ffb73a80251ba25b",
    "workspace": "/home/runner/work/MetaData/MetaData",
    "action": "sign-prov",
    "event_path": "/home/runner/work/_temp/_github_workflow/event.json",
    "action_repository": "",
    "action_ref": "",
    "path": "/home/runner/work/_temp/_runner_file_commands/add_path_94c8c796-8ef6-4c80-81[21](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241207367#step:6:21)-9f4a1f[39](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241207367#step:6:39)f8d3",
    "env": "/home/runner/work/_temp/_runner_file_commands/set_env_94c8c796-8ef6-4c80-8121-9f4a1f39f8d3",
    "step_summary": "/home/runner/work/_temp/_runner_file_commands/step_summary_94c8c796-8ef6-4c80-8121-9f4a1f39f8d3",
    "state": "/home/runner/work/_temp/_runner_file_commands/save_state_94c8c796-8ef6-4c80-8121-9f4a1f39f8d3",
    "output": "/home/runner/work/_temp/_runner_file_commands/set_output_94c8c796-8ef6-4c80-8121-9f4a1f39f8d3"
  }
Error: signing @: parsing reference: could not parse reference: @
main.go:[74](https://github.com/CHESSComputing/MetaData/actions/runs/7440318260/job/20241207367#step:6:74): error during command execution: signing @: parsing reference: could not parse reference: @vkuznet 
Error: Process completed with exit code 1.

Any clue from this output? As I already wrote my suspicious is that it is related to permission, as my github name is vkuznet and I'm working with repo under specific organization. May be I need specific rights under this organization in order to perform actions required by this workflow.

[jkreileder]

You try to set image and digest with:

          image_and_digest=$(echo "$ARTIFACTS" | jq -r '.[] | select (.type=="Docker Manifest") | .path')
          image=$(echo "${image_and_digest}" | cut -d'@' -f1 | cut -d':' -f1)
          digest=$(echo "${image_and_digest}" | cut -d'@' -f2)
          echo "name=$image" >> "$GITHUB_OUTPUT"
          echo "digest=$digest" >> "$GITHUB_OUTPUT"

But looking at your logs, e.g.

2024-01-08T13:41:19.7564015Z   ARTIFACTS: [{"name":"srv","path":"dist/MetaData_linux_arm64/srv","goos":"linux","goarch":"arm64","internal_type":4,"type":"Binary","extra":{"Binary":"srv","Ext":"","ID":"MetaData"}},{"name":"srv.exe","path":"dist/MetaData_windows_arm64/srv.exe","goos":"windows","goarch":"arm64","internal_type":4,"type":"Binary","extra":{"Binary":"srv","Ext":".exe","ID":"MetaData"}},{"name":"srv","path":"dist/MetaData_linux_amd64_v1/srv","goos":"linux","goarch":"amd64","goamd64":"v1","internal_type":4,"type":"Binary","extra":{"Binary":"srv","Ext":"","ID":"MetaData"}},{"name":"srv.exe","path":"dist/MetaData_windows_amd64_v1/srv.exe","goos":"windows","goarch":"amd64","goamd64":"v1","internal_type":4,"type":"Binary","extra":{"Binary":"srv","Ext":".exe","ID":"MetaData"}},{"name":"srv","path":"dist/MetaData_darwin_arm64/srv","goos":"darwin","goarch":"arm64","internal_type":4,"type":"Binary","extra":{"Binary":"srv","Ext":"","ID":"MetaData"}},{"name":"srv","path":"dist/MetaData_darwin_amd64_v1/srv","goos":"darwin","goarch":"amd64","goamd64":"v1","internal_type":4,"type":"Binary","extra":{"Binary":"srv","Ext":"","ID":"MetaData"}},{"name":"MetaData_Windows_arm64.zip","path":"dist/MetaData_Windows_arm64.zip","goos":"windows","goarch":"arm64","internal_type":1,"type":"Archive","extra":{"Binaries":["srv.exe"],"Checksum":"sha256:8f3aa7a72dcaa901c001e58d86507072cfd33b6b6984fe8e4398b1db23726216","Format":"zip","ID":"default","Replaces":null,"WrappedIn":""}},{"name":"MetaData_Darwin_arm64.tar.gz","path":"dist/MetaData_Darwin_arm64.tar.gz","goos":"darwin","goarch":"arm64","internal_type":1,"type":"Archive","extra":{"Binaries":["srv"],"Checksum":"sha256:f00afa13c70eb2f8d4f2c8d11d9e73aac433484a2d8ce8e178c571ec014da1c1","Format":"tar.gz","ID":"default","Replaces":null,"WrappedIn":""}},{"name":"MetaData_Windows_x86_64.zip","path":"dist/MetaData_Windows_x86_64.zip","goos":"windows","goarch":"amd64","goamd64":"v1","internal_type":1,"type":"Archive","extra":{"Binaries":["srv.exe"],"Checksum":"sha256:ee6daede95dd88b5dfb754f9ed7bc66e7898235a5f367f20f63ad6a112af7dfe","Format":"zip","ID":"default","Replaces":null,"WrappedIn":""}},{"name":"MetaData_Linux_x86_64.tar.gz","path":"dist/MetaData_Linux_x86_64.tar.gz","goos":"linux","goarch":"amd64","goamd64":"v1","internal_type":1,"type":"Archive","extra":{"Binaries":["srv"],"Checksum":"sha256:e9787f1b43d9ce320e24f0d636a5ad8272e2d1cfb47b4a989a945a719e896d73","Format":"tar.gz","ID":"default","Replaces":null,"WrappedIn":""}},{"name":"MetaData_Darwin_x86_64.tar.gz","path":"dist/MetaData_Darwin_x86_64.tar.gz","goos":"darwin","goarch":"amd64","goamd64":"v1","internal_type":1,"type":"Archive","extra":{"Binaries":["srv"],"Checksum":"sha256:784e9b64cb6e0c8f967976b321e80ae916ec92947eb231b32c3f34270dceecef","Format":"tar.gz","ID":"default","Replaces":null,"WrappedIn":""}},{"name":"MetaData_Linux_arm64.tar.gz","path":"dist/MetaData_Linux_arm64.tar.gz","goos":"linux","goarch":"arm64","internal_type":1,"type":"Archive","extra":{"Binaries":["srv"],"Checksum":"sha256:b6b0f2fa1c147496bd520b8fcbf2de14657acd308c828491b2b9981f0c178a03","Format":"tar.gz","ID":"default","Replaces":null,"WrappedIn":""}},{"name":"MetaData_0.0.0-pre2_checksums.txt","path":"dist/MetaData_0.0.0-pre2_checksums.txt","internal_type":12,"type":"Checksum","extra":{}}]

there is no .type=="Docker Manifest" in ARTIFACTS => image and digest are empty => generator_container_slsa3.yml fails

[ianlewis]

Indeed. It's a bit hard to understand. The generator build step always succeeds (continue-on-error==true) to support the continue-on-error input and the error is checked later in the final step. We could improve the error messages to make it easier for folks to know what happened and find the right log messages.

there is no .type=="Docker Manifest" in ARTIFACTS => image and digest are empty => generator_container_slsa3.yml fails

Right. This looks like the code copied from the blog post we link to in our docs and it looks like the image and digest are empty. We are trying to assemble it with "${UNTRUSTED_IMAGE}@${UNTRUSTED_DIGEST}". These values are empty so cosign is producing an error saying it doesn't know what to do with just the @ symbol. Again, we could improve the error messages here.

In the meantime could you try to debug this code a bit? We may need to update the docs if the newest version of the goreleaser action has changed.

[vkuznet]

thank you @jkreileder and @ianlewis for spotting the issue. Indeed, I was missing docker image build in my goreleaser. Once I added this step (via ko my build has finished without issue. Said that I'm fine to close this issue, but obviously documentation can be improved to avoid such mysterious message.

[laurentsimon]

This is being removed for v2 release. So closing