GCS publish: IAM_PERMISSION_DENIED during google-github-actions/auth

[jku]

This the cause of #63 which is a publish failure after the GCS publishing was first enabled: https://github.com/sigstore/root-signing-staging/actions/runs/8152961297/job/22283469446

Run google-github-actions/auth@a6e2e39c0a0331da29f7fd2c2a20a427e8d3ad1f
Created credentials file at "/home/runner/work/root-signing-staging/root-signing-staging/gha-creds-f5cc49c51c7cc2d8.json"
Error: google-github-actions/auth failed with: failed to generate Google Cloud OAuth 2.0 Access Token for tuf-gha@projectsigstore-staging.iam.gserviceaccount.com: {
  "error": {
    "code": 403,
    "message": "Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).",
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "IAM_PERMISSION_DENIED",
        "domain": "iam.googleapis.com",
        "metadata": {
          "permission": "iam.serviceAccounts.getAccessToken"
        }
      }
    ]
  }
}

[jku]

Production variables look like this:

workload_identity_provider: 'projects/306323169285/locations/global/workloadIdentityPools/github-actions-pool/providers/github-actions-provider'
service_account: 'github-actions@projectsigstore-staging.iam.gserviceaccount.com'

Here the values are in GitHub actions variables but otherwise used similarly:

 GCP_WORKLOAD_IDENTITY_PROVIDER: projects/306323169285/locations/global/workloadIdentityPools/github-actions-pool/providers/github-actions-provider
 GCP_SERVICE_ACCOUNT: tuf-gha@projectsigstore-staging.iam.gserviceaccount.com

So the service account is different but it looks correct.

[jku]

Oh the workload identity principal is: principal://iam.googleapis.com/projects/306323169285/locations/global/workloadIdentityPools/github-actions-pool/subject/repo:sigstore/root-signing-staging:ref:refs/heads/main

I think the issue is that we're likely not running on "refs/heads/main" (although I don't know exactly where google-github-actions/auth gets that value -- I assume some field in the JWT)

Longer term we could try to change that in tuf-on-ci if that's an issue... The original idea behind this was

• use a "publish" branch to document the currently publishable point on main: so the correct ref was refs/heads/publish: it's always safe to re-publish at any time from that branch (unlike main that could be unpublishable)
• but @kommendorkapten found out GitHub git repo and workflows are not always consistent: we can't push to publish branch and then expect to be able to trigger a workflow with the ref publish and expect to get the commit we just pushed (and we can't just wait for the push event because that doesn't trigger when using the default GH token)
• So he changed publish workflow so that it has a ref argument which is set to the specific commit revision when the workflow is dispatched (note that dispatch still happens on "publish")

[jku]

So my assumption here is that the ref in the workload identity principal is the ref the workflow runs on: This should be "publish" at the moment (see https://github.com/theupdateframework/tuf-on-ci/blob/main/actions/online-sign/action.yml#L90) .

Options:

1. We may be able to change the ref that the publish workflow runs on in tuf-on-ci without much actually changing: as mentioned above the real "ref-we-want-to-publish" is an input argument now anyway...
2. We could add refs/heads/publish to approved refs in the GCP workload id principal: "publish" is a protected branch so should be just as safe as refs/heads/main

Based on discussion with @kommendorkapten I'm leaning towards option 2 if that's feasible.

[haydentherapper]

Option (2) is in https://github.com/sigstore/public-good-instance/pull/2118.

I would like to hear more about the design choice to publish from a different branch than main. Is the concern that main ends up with invalid metadata?

[haydentherapper]

Applied 2118. We can follow up later if we want to change the ref later.

[jku]

I would like to hear more about the design choice to publish from a different branch than main. Is the concern that main ends up with invalid metadata?

I tried to do that in a comment above:

• "publish" branch documents the currently publishable point on main (right after merging a signing event, the HEAD of main is not something we want to publish. In practice "publish" moves whenever online-sign runs)
• There are a couple of complications:
  • we can't rely on push events on "publish" to actually trigger publish workflow (because when using default github token, that event will not be triggered by online-sign)
  • as Fredrik found out we also can't rely on dispatching publish workflow on ref "publish" after pushing to publish branch: Github systems are not consistent and the dispatch may actually run on an old branch at that point
• So in practice we currently
  • document the currently publishable point with branch "publish" -- this is still useful
  • dispatch publish workflow on the ref "publish" -- but this is not really anymore relevant and could be "main" as well
  • use an input argument to the workflow that actually specifies the commit we want to publish

[jku]

This one seems to work, debugging continues in #67