search

google-github-actions / auth

A GitHub Action for authenticating to Google Cloud.
https://cloud.google.com/iam
Apache License 2.0
949 stars 194 forks source link

Github actions 'Unable to acquire impersonated credentials' from GCP OIDC: error403 [principalSet mismatch with the Subject claim] #310

Closed brokedba closed 1 year ago

brokedba commented 1 year ago

TL;DR

Hi

Expected behavior

Observed behavior

everything seems to be working smoothly until the part where gcloud services list command run in the next step

image

Action YAML

name: 'Terraform_gcp_vpc'

on:
# ---snip-----
permissions:
  id-token: write

jobs: 
  terraform_setup:
    name: 'Terraform Init-Validate'
    runs-on: ubuntu-latest
    environment: gcp-labs
# ------ snip ------
    steps:
# Checkout the repository to the GitHub Actions runner
    - name: Checkout
      uses: actions/checkout@v3

 #  Authenticate with GCP  using OIDC Workload Federated Identity 
    - name: 'Authenticate to Google Cloud'
      id: auth
      uses: 'google-github-actions/auth@v1'
      with:
        workload_identity_provider: ${{ secrets.WORKLOAD_ID_PROVIDER }} 
        service_account: ${{ secrets.SERVICE_ACCOUNT }}

  # Install gcloud, `setup-gcloud` automatically picks up authentication from `auth`.
    - name: 'Set up Cloud SDK'
      uses: 'google-github-actions/setup-gcloud@v1'   
    - id: 'gcloud'
      name: 'gcloud'
      run: gcloud services list

Log output

Error: (gcloud.services.list) There was a problem refreshing your current auth tokens: 'Unable to acquire impersonated credentials'
 {
  "error": {
    "code": 403,
    "message": "Permission \'iam.serviceAccounts.getAccessToken\' denied on resource (or it may not exist).",
    "status": "PERMISSION_DENIED"
    "details":
   [
      {
      "reason": "IAM_PERMISSION_DENIED",        
      "domain": "iam.googleapis.com",
      "metadata": {
      "permission": "iam.serviceAccounts.getAccessToken"
      }
    }
   ]
  }
 }

Please run:

  $ gcloud auth login
to obtain new credentials.

Additional information

Here's gist of all the steps taken to set it up including audit log if it can help 👉🏽: https://bit.ly/3IIONjC

Attribute-Mapping

image

PrincipalSet

principalSet://iam.googleapis.com/${WORKLOAD_IDENTITY_POOL_ID}/attribute.repository/${REPO}:environment:gcp-labs
github-actions[bot] commented 1 year ago

Hi there @brokedba :wave:!

Thank you for opening an issue. Our team will triage this as soon as we can. Please take a moment to review the troubleshooting steps which lists common error messages and their resolution steps.

brokedba commented 1 year ago

Is this sub assertion correct ?:

repo:<orgName/repoName>:ref:refs/heads/branchName:environment:staging
"sub": "repo:<orgName/repoName>:ref:refs/heads/branchName:environment:<environmentName>"
"claims_supported":["sub","aud","exp","iat","iss","jti","nbf","ref","repository","repository_id","repository_owner","repository_owner_id","run_id","run_number","run_attempt","actor","actor_id","workflow","workflow_ref","workflow_sha","head_ref","base_ref","event_name","ref_type","environment","environment_node_id","job_workflow_ref","job_workflow_sha","repository_visibility","runner_environment"],"id_token_signing_alg_values_supported":

image

sethvargo commented 1 year ago
  • which I confirmed the structure from the official documentation

Please use the GitHub Actions OIDC debugger to verify the claims are what you expect. We've experienced situations where the documentation is incorrect.

Action YAML

Can you please provide an action.yaml that uses the steps outlined in the TROUBLESHOOTING guide? Does adding token_format: 'access_token' cause the GitHub Action to error at the "auth" step?

- name: 'Authenticate to Google Cloud'
  id: auth
  uses: 'google-github-actions/auth@v1'
  with:
    workload_identity_provider: ${{ secrets.WORKLOAD_ID_PROVIDER }} 
    service_account: ${{ secrets.SERVICE_ACCOUNT }}
    token_format: 'access_token' # <-- add this

Please also verify that the secrets are actually being injected into the environment. Since neither of these values are actually "secret", I would recommend using GitHub variables instead of GitHub secrets. For the purposes of debugging, you can also hardcode them in the YAML.

It's really difficult to follow what you're actually trying to do here between the screenshots and abstractions with secrets and environment variables.

brokedba commented 1 year ago

Thank you @sethvargo for the reply . please refer to the gist for the steps I followed . Here is it including audit log if it can help 👉🏽: https://bit.ly/3IIONjC

It's really difficult to follow what you're actually trying to do here between the screenshots and abstractions with secrets and environment variables.

I am just trying to terraform deploy gcp resources through my github actions and OIDC and WIF provider. I did that with other cloud providers (azure/aws) and it seamless and easy to implement. I didn't find the doc helpful so far after 2 weeks of struggling. Mapping and principal sets just don't recognize my repo token.

Please also verify that the secrets are actually being injected into the environment. Since neither of these values are actually "secret", I would recommend using GitHub variables instead of GitHub secrets.

Yes they are if we check the yaml env section at jobs level

sethvargo commented 1 year ago

Can you please answer the 3 other questions? What does the OIDC debugger show as the claims? What happens when you add token_format: 'access_token'? Please verify that the secret is actually being injected.

brokedba commented 1 year ago

Can you please answer the 3 other questions? What does the OIDC debugger show as the claims? What happens when you add token_format: 'access_token'? Please verify that the secret is actually being injected.

Error: Input required and not supplied: token
I think I didn't adapt the template action with the actual token . Should I add a token from the runner's environment instead of secret here ?

    - name: Checkout
      uses: actions/checkout@v3
      with:
        repository: github/actions-oidc-debugger
        ref: main
        token: ${{ secrets.your-checkout-token }} <<<<
        path: ./.github/actions/actions-oidc-debugger
    - name: Debug OIDC Claims
      uses: ./.github/actions/actions-oidc-debugger
      with:
        audience: 'https://github.com/github'

when I add token_format: 'access_token' I get the following error 

Created credentials file at "/home/runner/work/terraform-examples/terraform-examples/gha-creds-90d9a8ea8b9cff99.json"
Created credentials file at "/home/runner/work/terraform-examples/terraform-examples/gha-creds-cc67cc6c5d31e67a.json"
Warning: Overwriting existing environment variable CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE:
...
Error: google-github-actions/auth failed with: retry function failed after 4 attempts: failed to generate Google Cloud access token for ***: (403) {..
  "error": {
    "code": 403,
    "message": "Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).",
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "IAM_PERMISSION_DENIED",
        "domain": "iam.googleapis.com",
        "metadata": {
          "permission": "iam.serviceAccounts.getAccessToken"
        }

please find the workflow run log here :

brokedba commented 1 year ago

It is really really frustrating. I Knew the problem came from the Mapping attributes vs Principalset claim Paterns mismatch . But no where in the doc did GCP help developers to set the right assertions and PrincipalSet definition . WHY IS THAT ? I had to look for youtube videos and blogs to seek for answers because this simple subject use case "sub": "repo:<orgName/repoName>:ref:refs/heads/branchName:environment:<environmentName>" couldn't be explained in the documentation . I lost 2 weeks on this .

sethvargo commented 1 year ago

Hi @brokedba - awesome. I'm glad you were able to figure it out!

brokedba commented 1 year ago

@sethvargo please keep the issue open as it's not fixed at all. I wanted to use environment in the attribute mapping and it's still not working . I'll consider this issue resolved if I get the answer to hat .

brokedba commented 1 year ago

My OP's issue is I can't get my repo ID to be recognized using following claim :

"sub": "repo:<orgName/repoName>:environment:<environmentName>"

I'm not satisfied with the answer yet as repository attribute I used is too broad . I then tried the below concatenation which didn't work [403 error]:

1. provider 

gcloud iam workload-identity-pools providers create-oidc "gitaction-oidc-provider" --project="${PROJECT_ID}" \
--location="global" --workload-identity-pool="github-pool" --display-name="gitactions-oidc-provider" \
 --attribute-mapping="google.subject=assertion.sub,attribute.full=assertion.repository+assertion.environment \
  --issuer-uri="https://token.actions.githubusercontent.com/"

2. Principalset

gcloud iam service-accounts add-iam-policy-binding "$SA" \
--project="${PROJECT_ID}" --role="roles/iam.workloadIdentityUser" \
--member="principalSet://[iam.googleapis.com](http://iam.googleapis.com/)/${WORKLOAD_IDENTITY_POOL_ID}/attribute.full/${REPO}:environment:gcp-labs"
sethvargo commented 1 year ago

Here is what I meant by trying to reproduce the issue with the smallest number of steps:

name: 'repro'

on:
  push:

permissions:
  contents: 'read'
  id-token: 'write'

jobs:
  job:
    runs-on: 'ubuntu-latest'
    steps:
      - uses: 'actions/checkout@v3'

      - uses: 'github/actions-oidc-debugger@main'
        with:
          audience: 'https://iam.googleapis.com/projects/99343382058/locations/global/workloadIdentityPools/github/providers/github'

      - id: 'auth'
        uses: 'google-github-actions/auth@main'
        with:
          workload_identity_provider: 'projects/99343382058/locations/global/workloadIdentityPools/github/providers/github'
          service_account: 'external@poopy-candles.iam.gserviceaccount.com'
          token_format: 'access_token'

This will print out the OIDC token claims and attempt to generate a token. As I expected, the value for sub does not always match what is in GitHub's documentation. GitHub's documentation says the value for sub is repo:octo-org/octo-repo:environment:prod, but when I used the GitHub Actions OIDC debugger, the result was repo:sethvargo-demos/ghactions-test:ref:refs/heads/main. The sub can undertake many different values and formats, so the first thing to check is to make sure that the subject is the value that you're expecting. Note, this is how GitHub works and has nothing to do with Google Cloud.

I was able to get this working fine. Here's my pool configuration:

provider-config

And here's the Service Account permission which has Workload Identity User permissions.

principal://iam.googleapis.com/projects/99343382058/locations/global/workloadIdentityPools/github/subject/repo:sethvargo-demos/ghactions-test:ref:refs/heads/main
brokedba commented 1 year ago

Thank you Seth , I will keep the debugger yaml template handy in the future . Your example didn't include the environment in the sub claim probably because your don't have one in your repo . I think the default google.subject inferred expression is still not clear or doesn't include environment cases .

sethvargo commented 1 year ago

Hi @brokedba

I don't know what you mean by:

I think the default google.subject inferred expression is still not clear or doesn't include environment cases.

  1. Hope you can agree on the contradiction between the 2 src , one says repo+branch+environment the latter only repo+branch:

You need to click on the link in the GitHub docs (in your screenshot) that says "Example subject claims". The subject claim can take 5 different formats (as of the time of this writing), and GitHub is always adding more. GitHub controls the subject they inject into the token, we do not. The subject format and value is both a function of the repository configuration and a function of how the workflow is triggered (e.g. push, PR, deploy). These are all GitHub concepts.

The Google Cloud documentation says "for example" repo:username/reponame:ref:refs/head/master. The reason there's a link to the GitHub documentation at the very top of that box is so that users can read the canonical source of information about the token claims from GitHub. I don't think the GCP documentation team is going to import all of GitHub's documentation on OIDC tokens into this box.

Finally, as shown above, sometimes it is only repo+branch (if you don't have an environment). This is why it's important to read the GitHub documentation on the possible subject claim values.

brokedba commented 1 year ago
sethvargo commented 1 year ago

Hi @brokedba at no point did I say you did not read the documentation. You commented that the GCP documentation was incorrect, which I disagreed with. I said that the GCP documentation lists an example of what the sub might be, and that GitHub provides an authoritative list. I also noted that the sub varies based on how the GitHub Action is triggered.

Here's a working end-to-end example (it's the same I posted above, but I made the repo public): https://github.com/sethvargo-demos/ghactions-test/actions/runs/5126239568/jobs/9220483047

The only mappings are google.subject=assertion.sub and the principalSet mapping is for the value that is shown in the OIDC debugger ("repo:sethvargo-demos/ghactions-test:environment:prod"):

principal://iam.googleapis.com/projects/99343382058/locations/global/workloadIdentityPools/github/subject/repo:sethvargo-demos/ghactions-test:environment:prod

These are also the exact steps I described above.

brokedba commented 1 year ago

Hi @sethvargo , In the meantime I found a solution to get my repo authenticated using a provider custom attribute, with a little help from @amasucci . I used the below custom attribute and assertion concatenation (repo+environment)

Either way , this concatenation does the job too so we can consider this issue solved. I'll keep you posted tomorrow morning.

thanks

sethvargo commented 1 year ago

You are missing the literal string repo: in front of the mapping.

Mine:

principal://iam.googleapis.com/${POOL_ID}/subject/repo:${REPO}:environment:prod

Yours:

principalSet://iam.googleapis.com/${POOL_ID}/subject/${REPO}:environment:gcp-labs

As I said above, you need to take the complete literal string that the OIDC debugger returns and use that as your attribute mapping.

-principalSet://iam.googleapis.com/${POOL_ID}/subject/${REPO}:environment:gcp-labs
+principalSet://iam.googleapis.com/${POOL_ID}/subject/repo:${REPO}:environment:gcp-labs
brokedba commented 1 year ago

My attribute based mapping (attribute.full) is fine but you are correct Seth regarding the subject based mapping . I tried it yesterday and I ought to update the issue .

This is what my debugger replied with : 

{
  "actor": "brokedba",
  "aud": "https://github.com/brokedba",
  "environment": "gcp-labs",
  "event_name": "push",
   "iss": "https://token.actions.githubusercontent.com",
 "job_workflow_ref": "brokedba/terraform-examples/.github/workflows/terraform_gcp-labs_vpc.yml@refs/heads/git_actions",
  "ref": "refs/heads/git_actions",
  "ref_type": "branch",
  "repository": "brokedba/terraform-examples",
  "repository_owner": "brokedba",
snip ...
 "sub": "repo:brokedba/terraform-examples:environment:gcp-labs",  <<-----
 "workflow": "Terraform_gcp_vpc",
 "workflow_ref": "brokedba/terraform-examples/.github/workflows/terraform_gcp-labs_vpc.yml@refs/heads/git_actions",
  }

Forgive my ignorance as I digged through GCP docs but haven't found an article/doc explaining clear differences between Principal VS PrincipalSet . .

I'm going to be a bit verbose this time as it's my closing update to this issue in case anyone ends up in a similar spot.

You can confirm whether this summary I collected here and there from blogs/medium etc is correct:

Principals

"principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/YOUR_PROJECT_ID/attribute.repository/${REPO}"
 - note that the URI scheme is now principalSet instead of principal 
 - with these attribute you can set up some very fine grained controls to allow access to specific service accounts from specific events on specific repositories.
laysauchoa commented 5 months ago

Hi @brokedba I am facing the same issue and could not find good direction on how to resolve it. Were you able to solve it by using repository and environment restrictions? If yes, can you post what your mapping and principalset look like? Thanks in advance.

brokedba commented 5 months ago

@laysauchoa , It's been almost a year now and I don't have view on the gcp console , but if you read the output of my debugger , you can see the below 

"sub": "repo:brokedba/terraform-examples:environment:gcp-labs",  <<-----

I'd suggest not to use principalset based configuration and stick with Principal . if your case involves repository and environment restrictions. your expression would look like the below 

"principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/YOUR_PROJECT_ID/subject/repo:${REPO}:environment:myenvironement"`
example:
principal://iam.googleapis.com/projects/xx/locations/global/workloadIdentityPools/mypool/subject/repo:user/repoName:environment:My-env
laysauchoa commented 5 months ago

Thank you so much @brokedba