Failed to generate Google Cloud ID token for service_account

[VikramTiwari]

TL;DR

I am getting an auth bug when I try to generate an id_token using the github action.

Expected behavior

There should be no auth errors.

Observed behavior

I get a 403 error.

Run google-github-actions/auth@v0
  with:
    token_format: id_token
    workload_identity_provider: projects/XXXX/locations/global/workloadIdentityPools/ABCD/providers/ABCD-provider
    service_account: abcd@***.iam.gserviceaccount.com
    id_token_audience: projects/XXXX/locations/global/workloadIdentityPools/ABCD/providers/ABCD-provider
    create_credentials_file: true
    cleanup_credentials: true
    access_token_lifetime: 3600s
    access_token_scopes: https://www.google***s.com/auth/cloud-platform
    id_token_include_email: false
  env:
    WORKLOAD_IDENTITY_POOL_PROVIDER: projects/XXXX/locations/global/workloadIdentityPools/ABCD/providers/ABCD-provider
    SERVICE_ACCOUNT: abcd@***.iam.gserviceaccount.com
Created credentials file at "/home/runner/work/***/***/3eb53b95d764be1564f71360"

Error: google-github-actions/auth failed with: failed to generate Google Cloud ID token for abcd@***.iam.gserviceaccount.com: {
  "error": {
    "code": 403,
    "message": "The caller does not have permission",
    "status": "PERMISSION_DENIED"
  }
}

Action YAML

name: Deploy
on:
  push:
    branches:
      - 'develop'

jobs:
  setup-build-publish-deploy:
    name: Setup, Build, Publish, and Deploy
    runs-on: ubuntu-latest

    env:
      WORKLOAD_IDENTITY_POOL_PROVIDER: 'projects/XXXX/locations/global/workloadIdentityPools/ABCD/providers/ABCD-provider'
      SERVICE_ACCOUNT: 'abcd@omni-daeos.iam.gserviceaccount.com'

    permissions:
      contents: 'read'
      id-token: 'write'

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Build Docker image
        run: |
          docker build . --tag gcr.io/${{ secrets.GCLOUD_PROJECT }}/${{ secrets.GCLOUD_APP_NAME }}-v${{ steps.package-version.outputs.current-version }}

      # Configure Workload Identity Federation for Google Cloud
      - id: auth
        name: 'Authenticate to Google Cloud'
        uses: 'google-github-actions/auth@v0'
        with:
          token_format: 'id_token'
          workload_identity_provider: ${{ env.WORKLOAD_IDENTITY_POOL_PROVIDER }}
          service_account: ${{ env.SERVICE_ACCOUNT }}
          id_token_audience: ${{ env.WORKLOAD_IDENTITY_POOL_PROVIDER }}
          create_credentials_file: true

      # Setup gcloud CLI
      - name: Setup gcloud sdk
        uses: google-github-actions/setup-gcloud@v0

      # Configure docker to use the gcloud command-line tool as a credential helper
      - name: 'Set up docker to authenticate via gcloud command-line tool'
        run: |
          gcloud auth login --brief --cred-file="${{ steps.auth.outputs.credentials_file_path }}"
          gcloud auth configure-docker

      - name: Push Docker image to Google Container registry
        run: |
          docker push gcr.io/${{ secrets.GCLOUD_PROJECT }}/${{ secrets.GCLOUD_APP_NAME }}-v${{ steps.package-version.outputs.current-version }}

Additional information

No response

[VikramTiwari]

The issue isn't from github repo itself but I still checked the bindings to ensure everything is proper there. Here's the result:

vikram@cloudshell:~ (PROJECT_ID)$ gcloud iam service-accounts get-iam-policy "abcd@${PROJECT_ID}.iam.gserviceaccount.com"
bindings:
- members:
  - principalSet://iam.googleapis.com/projects/XXXX/locations/global/workloadIdentityPools/ABCD/attribute.repository/omnilabsinc/xxx
  - principalSet://iam.googleapis.com/projects/XXXX/locations/global/workloadIdentityPools/ABCD/attribute.repository_owner/omnilabsinc
  role: roles/iam.workloadIdentityUser
etag: BwXUs6cVGrA=
version: 1

[sethvargo]

Hi @VikramTiwari

Thank you for opening an issue. A few questions:

1. Why are you trying to generate an ID token? I don't see you consuming that output in any other steps. You do not need to generate a token if you don't plan on using it for future steps.

2. If you're trying to avoid using gcloud to configure Docker, you need to pass an access token, not an ID token to Docker (docs).

3. Did you set map those claims in the Workload Identity Provider? You must map repository and repository_owner from the provider:

   

4. Have you waited at least 5 minutes since the configuration was created? Changing WIF attributes and IAM permissions are eventually consistent.

5. Have you checked the logs for the STS endpoint? You can check the logs with the following query:

   resource.type="audited_resource" resource.labels.service="sts.googleapis.com" protoPayload.resourceName:"workloadIdentityPools/YOUR_POOL_NAME_HERE"

[VikramTiwari]

Thanks for the detailed answer and suggestions @sethvargo. Here are some comments:

1. Why are you trying to generate an ID token? I don't see you consuming that output in any other steps. You do not need to generate a token if you don't plan on using it for future steps.

I was originally trying to not generate any token and use the action as is. But I kept running into 403s just before publish. Here's the error:

Run docker push gcr.io/***/***-v0.X.X
Using default tag: latest
ERROR: (gcloud.auth.docker-helper) There was a problem refreshing your current auth tokens: ('Unable to acquire impersonated credentials: No access token or invalid expiration in response.', '{\n  "error": {\n    "code": 403,\n    "message": "The caller does not have permission",\n    "status": "PERMISSION_DENIED"\n  }\n}\n')
Please run:

  $ gcloud auth login

to obtain new credentials.

If you have already logged in with a different account:

    $ gcloud config set account ACCOUNT

to select an already authenticated account to use.
The push refers to repository [gcr.io/***/***-v0.X.X]
0e6f80d1bb7c: Preparing
c33fedfc3c07: Preparing
e510497f0ea9: Preparing
05bfcb76cbbc: Preparing
df329b99df52: Preparing
0722c4b9358c: Preparing
214f1f0bcd56: Preparing
37fd5d56f1d2: Preparing
fbaccaf77177: Preparing
be2ae69cd530: Preparing
1a058d5342cc: Preparing
0722c4b9358c: Waiting
214f1f0bcd56: Waiting
37fd5d56f1d2: Waiting
fbaccaf77177: Waiting
be2ae69cd530: Waiting
1a058d5342cc: Waiting
unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
Error: Process completed with exit code 1.

2. If you're trying to avoid using gcloud to configure Docker, you need to pass an access token, not an ID token to Docker (docs).

I am not trying to avoid this. My previous workflow was just to authenticate gcloud with service account and then use gcloud auth configure-docker to configure docker and then rest work without anything else. However, for this, I have updated the settings to generate access_token now.

3. Did you set map those claims in the Workload Identity Provider? You must map repository and repository_owner from the provider:

Confirming that these are mapped properly.

4. Have you waited at least 5 minutes since the configuration was created? Changing WIF attributes and IAM permissions are eventually consistent.

Yes. I have even run it more than once with more than an hour of time different between changes.

5. Have you checked the logs for the STS endpoint? You can check the logs with the following query:

   resource.type="audited_resource" resource.labels.service="sts.googleapis.com" protoPayload.resourceName:"workloadIdentityPools/YOUR_POOL_NAME_HERE"

I don't see anything in the logs with this query, after replacing with my pool id there. Even after using resource.type="audited_resource" resource.labels.service="sts.googleapis.com" query.

I am sure there's something basic I am missing.

[sethvargo]

Hi @VikramTiwari

Thank you for the reply. Without access to the environment, it's really difficult to debug WIF. What is the output of:

gcloud iam workload-identity-pools providers describe YOUR_PROVIDER_NAME \
  --project=YOUR_PROJECT_ID \
  --location=global \
  --workload-identity-pool=YOUR_POOL

[camelpunch]

Having similar issues using the auth action preceding a docker push, but mapping the repository attribute fixed it for us. Still, we had to use the workaround of piping the token into docker login, rather than using the setup-gcloud action.

[sethvargo]

@camelpunch if you're just trying to push to a Docker registry like Container Registry or Artifact Registry, you can skip setup-gcloud and similar steps:

jobs:
  run:
    # ...

    # Add "id-token" with the intended permissions.
    permissions:
      id-token: write
      contents: read

    steps:
    - id: 'gcp-auth'
      name: 'Authenticate to Google Cloud'
      uses: 'sethvargo/oidc-auth-google-cloud@v0'
      with:
        token_format: 'access_token'
        workload_identity_provider: '[REPLACE]'
        service_account: '[REPLACE]'

    - id: 'docker-login'
      run: |-
        echo "${{ steps.gcp-auth.outputs.access_token }}" | docker login -u oauth2accesstoken --password-stdin https://[REGION]-docker.pkg.dev

If you're using a Docker action, the username is "oauth2accesstoken" and the password is the token

[camelpunch]

Yes, we followed that advice from another issue. Thanks!

[sethvargo]

I'm going to close this due to inactivity. Vikram - if this continues to not work, please open a new issue and provide all the required fields in our issue template. Thanks!