Caller does not have required permission to use project

[GergelyKalmar]

TL;DR

I cannot seem to be able to get the application default credentials to work with the Python SDK.

Expected behavior

I expected the following code to run properly:

from google.auth import default
from google.auth.transport.requests import AuthorizedSession

credentials = default(quota_project_id='xxx-yyy')[0]
session = AuthorizedSession(credentials)
response = session.get('https://www.googleapis.com/oauth2/v3/tokeninfo')
print(response.json())

Observed behavior

I get the following error:

RefreshError: ('Unable to acquire impersonated credentials', '{
  "error": {
    "code": 403,
    "message": "Caller does not have required permission to use project xxx-yyy.
Grant the caller the roles/serviceusage.serviceUsageConsumer role,
or a custom role with the serviceusage.services.use permission,
by visiting https://console.developers.google.com/iam-admin/iam/project?project=xxx-yyy and then retry.
Propagation of the new permission may take a few minutes.",
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.Help",
        "links": [
          {
            "description": "Google developer console IAM admin",
            "url": "https://console.developers.google.com/iam-admin/iam/project?project=xxx-yyy"
          }
        ]
      },
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "USER_PROJECT_DENIED",
        "domain": "googleapis.com",
        "metadata": {
          "service": "iamcredentials.googleapis.com",
          "consumer": "projects/xxx-yyy"
        }
      }
    ]
  }
}
')

The Python SDK logs show the following:

------------------------------ Captured log call -------------------------------
2022-12-17 14:14:41.496 DEBUG Checking /home/runner/work/xxx/gha-creds-xxx.json for explicit credentials as part of auth process... (google.auth._default:218)
2022-12-17 14:14:41.509 DEBUG This service is instrumented using OpenTelemetry. OpenTelemetry or one of its components could not be imported; please add compatible versions of opentelemetry-api and opentelemetry-instrumentation packages in order to get BigQuery Tracing data. (google.cloud.bigquery.opentelemetry_tracing:66)
2022-12-17 14:14:41.509 DEBUG Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None) (urllib3.util.retry:351)
2022-12-17 14:14:41.510 DEBUG Making request: GET https://pipelines.actions.githubusercontent.com/xxx/idtoken?api-version=2.0&audience=xxx (google.auth.transport.requests:192)
2022-12-17 14:14:41.516 DEBUG Starting new HTTPS connection (1): pipelines.actions.githubusercontent.com:443 (urllib3.connectionpool:1003)
2022-12-17 14:14:41.603 DEBUG https://pipelines.actions.githubusercontent.com:443 "GET xxx/idtoken?api-version=2.0&audience=xxx HTTP/1.1" 200 None (urllib3.connectionpool:456)
2022-12-17 14:14:41.604 DEBUG Making request: POST https://sts.googleapis.com/v1/token (google.auth.transport.requests:192)
2022-12-17 14:14:41.607 DEBUG Starting new HTTPS connection (1): sts.googleapis.com:443 (urllib3.connectionpool:1003)
2022-12-17 14:14:41.703 DEBUG https://sts.googleapis.com:443 "POST /v1/token HTTP/1.1" 200 None (urllib3.connectionpool:456)
2022-12-17 14:14:41.705 DEBUG Making request: POST https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/xxx.iam.gserviceaccount.com:generateAccessToken (google.auth.transport.requests:192)
2022-12-17 14:14:41.710 DEBUG Starting new HTTPS connection (1): iamcredentials.googleapis.com:443 (urllib3.connectionpool:1003)
2022-12-17 14:14:41.824 DEBUG https://iamcredentials.googleapis.com:443 "POST /v1/projects/-/serviceAccounts/xxx.iam.gserviceaccount.com:generateAccessToken HTTP/1.1" 403 None (urllib3.connectionpool:456)

Action YAML

It is a bit more complex because the errors are appearing in a private test suite, however, based on the information above it should be possible to figure out what the issue is.

Log output

No response

Additional information

I have tried adding the Service Usage Consumer role to the impersonated service account, it did not help. The authorization otherwise works and I can also run the following action after the auth:

      - run: |-
          curl https://www.googleapis.com/oauth2/v3/tokeninfo \
            --header "Authorization: Bearer ${{ steps.auth.outputs.access_token }}"

This shows

curl https://www.googleapis.com/oauth2/v3/tokeninfo \
    --header "Authorization: ***"
  shell: /usr/bin/bash -e {0}
  env:
    CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /home/runner/work/.../gha-creds-xxx.json
    GOOGLE_APPLICATION_CREDENTIALS: /home/runner/work/.../gha-creds-xxx.json
    GOOGLE_GHA_CREDS_PATH: /home/runner/work/.../gha-creds-xxx.json
    CLOUDSDK_CORE_PROJECT: xxx-yyy
    CLOUDSDK_PROJECT: xxx-yyy
    GCLOUD_PROJECT: xxx-yyy
    GCP_PROJECT: xxx-yyy
    GOOGLE_CLOUD_PROJECT: xxx-yyy
...
{
  "azp": "xxx",
  "aud": "xxx",
  "scope": "https://www.googleapis.com/auth/cloud-platform",
  "exp": "89837",
  "expires_in": "3599",
  "access_type": "online"
}

Any advice would be highly appreciated.

[github-actions[bot]]

Hi there @GergelyKalmar :wave:!

Thank you for opening an issue. Our team will triage this as soon as we can. Please take a moment to review the troubleshooting steps which lists common error messages and their resolution steps.

[GergelyKalmar]

One more thing to note: the error is raised at .../lib/python3.8/site-packages/google/auth/impersonated_credentials.py:103: RefreshError

[sethvargo]

Hi @GergelyKalmar

Why are you setting quota project? If you set quota project, then the calling principal must have roles/serviceusage.serviceUsageConsumer permissions on the quota project.

[GergelyKalmar]

hi @sethvargo, that's mostly because I want to force the code to use a given project for quotas even if the user's application default credentials happen to be using a different quota project (when running locally).

Do we know who the "calling principal" is? I'm asking because even if I attach the roles/serviceusage.serviceUsageConsumer to the service account (on the quota project, xxx-yyy in this hypothetical case) it still does not work. In fact, I can make the service user the owner of the quota project and I'm still getting the same error.

[sethvargo]

Hi @GergelyKalmar - it would be whatever service account you're authenticating as. When you sey a quota project, the calling identity must have roles/serviceusage.serviceUsageConsumer on the target quota project (even if it's the same destination or "origin" project as a service account). Also note that propagation may take up to 5 minutes for IAM permission changes.

You didn't provide an action.yaml or debug logs from the GitHub Actions runner, so it's difficult to try and build a reproduction case. Does the issue still occur if you use gcloud directly and then use the Python SDK:

gcloud auth login --update-adc --cred-file /path/to/wif.json

And then run your Python script locally. If the issue still persists, that would indicate a problem with your permissions or potentially a bug in the Python SDK.

[GergelyKalmar]

Hm, interestingly if I do remove the quota project ID everything seems to be working.

I also tried to impersonate the service account using

gcloud auth application-default login --impersonate-service-account=xxx.iam.gserviceaccount.com --configuration=yyy

and then ran the test suite locally, again, everything worked, even when the quota project ID was specified. It's all really puzzling. Can I somehow generate a similar ADC locally as what this action generates? It seems quite difficult to get the file out of the GitHub Actions runner safely for local testing.

[sethvargo]

Hi @GergelyKalmar

--impersonate-service-account is only persistent for that call. If you want it to persist for all API calls, you need to set it in the config:

gcloud config set auth/impersonate_service_account foo@project.iam.gserviceaccount.com

The easiest way to reproduce would be to download the WIF credential file from the web UI (or use gcloud iam workload-identity-pools create-cred-config).

You don't need to specify a quota project when the destination project is the same as the quota target project. In fact, it's not recommended that you do that, since then you'll need the serviceUsage permissions on that project.

[GergelyKalmar]

gcloud auth login --update-adc --cred-file /path/to/wif.json

And then run your Python script locally. If the issue still persists, that would indicate a problem with your permissions or potentially a bug in the Python SDK.

I'm trying to get there, however, I wasn't really able to make this work. Where would I need to download the WIF credential file and what would be the configuration settings for GitHub Actions in particular? (Either in the UI or when using gcloud iam workload-identity-pools create-cred-config.)

[sethvargo]

https://cloud.google.com/iam/docs/using-workload-identity-federation#console_1

[GergelyKalmar]

Indeed that's what I tried, however, I'm not quite sure what settings to use here:

The settings I guessed ended up with this file:

{
  "type": "external_account",
  "audience": "//iam.googleapis.com/projects/XXX/locations/global/workloadIdentityPools/XXX/providers/github-actions",
  "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
  "token_url": "https://sts.googleapis.com/v1/token",
  "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/xxx@yyy.iam.gserviceaccount.com:generateAccessToken",
  "credential_source": {
    "url": "https://token.actions.githubusercontent.com",
    "headers": {},
    "format": {
      "type": "json",
      "subject_token_field_name": "access_token"
    }
  }
}

The error I got is this:

google.auth.exceptions.RefreshError: ('Unable to retrieve Identity Pool subject token', '')

Logs:

2022-12-20 22:10:08.251 DEBUG Checking None for explicit credentials as part of auth process... (google.auth._default:218)
2022-12-20 22:10:08.252 DEBUG Checking Cloud SDK credentials as part of auth process... (google.auth._default:191)
...
2022-12-20 22:10:11.504 DEBUG Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None) (urllib3.util.retry:351)
2022-12-20 22:10:11.505 DEBUG Making request: GET https://token.actions.githubusercontent.com (google.auth.transport.requests:192)
2022-12-20 22:10:11.509 DEBUG Starting new HTTPS connection (1): token.actions.githubusercontent.com:443 (urllib3.connectionpool:1003)
2022-12-20 22:10:11.971 DEBUG https://token.actions.githubusercontent.com:443 "GET / HTTP/1.1" 404 0 (urllib3.connectionpool:456)

[sethvargo]

Do you have access to the service account xxx@yyy.iam.gserviceaccount.com? It might be easier just download the JSON service account key for that service account than to try and get WIF working locally.

[GergelyKalmar]

Yes, that wouldn't really test the same flow though, or would it? I tend to believe that this is a bug in the Python SDK as you suggested (given that the action works as expected otherwise), however, I don't really have the time to work on a reproducer myself, so I think I'll close the issue.

If you wanted to, you could just take the code snippet I wrote, put it into a Python file and run it in GitHub Actions (after checkout, this auth action and once the google-auth requirement is installed).