Closed selfagency closed 2 years ago
@sethvargo We're totally blocked here so it would be greatly appreciated if you could find time this morning to help me out. Thanks.
Hi @selfagency
Can you try adding "token_format: access_token" to the auth
step temporarily? If the Action proceeds beyond the auth step, it means there's an issue with the upstream service. However, if the Action fails at the auth step (after adding that token), it means the issue is in the authentication handshake, and the specific error message will give us more information.
@sethvargo
##[debug]Evaluating condition for step: 'Login to Google Cloud'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Login to Google Cloud
##[debug]Register post job cleanup for action: google-github-actions/auth@v0
##[debug]Loading inputs
##[debug]Evaluating: secrets.WORKLOAD_ID_PROVIDER_STAGING
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'WORKLOAD_ID_PROVIDER_STAGING'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Evaluating: secrets.SERVICE_ACCOUNT_STAGING
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'SERVICE_ACCOUNT_STAGING'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Loading env
Run google-github-actions/auth@v0
##[debug]Using workload identity provider "***"
##[debug]ID token url is https://pipelines.actions.githubusercontent.com/JIdNBcQ6kMWGQ1Emvk4eWkVClMzQPdc1GeXypChXNSBsqbvPVO/00000000-0000-0000-0000-000000000000/_apis/distributedtask/hubs/Actions/plans/bedb98a3-f043-4f9e-88[2](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:2)7-c75a2c08e90b/jobs/a9[3](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:3)157a9-a0f6-5af8-a70b-8506ae[4](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:4)7[5](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:5)1b2/idtoken?api-version=2.0&audience=https%3A%2F%2Fiam.googleapis.com%2F***
::add-mask::***
##[debug]Creating credentials file
Created credentials file at "/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-9007291eaec30215.json"
::set-output name=credentials_file_path::/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-9007291eaec30215.json
##[debug]steps.auth.outputs.credentials_file_path='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-9007291eaec30215.json'
::set-output name=project_id::unstoppable-domains-staging
##[debug]steps.auth.outputs.project_id='unstoppable-domains-staging'
Warning: Overwriting existing environment variable GCP_PROJECT (was: "unstoppable-domains-staging")
##[debug]Creating access token
::add-mask::***
::set-output name=access_token::***
##[debug]steps.auth.outputs.access_token='***'
::set-output name=access_token_expiration::2022-04-05T18:4[6](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:6):56Z
##[debug]steps.auth.outputs.access_token_expiration='2022-04-05T18:46:56Z'
##[debug]Node Action run completed with exit code 0
##[debug]CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-900[7](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:7)2[9](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:9)1eaec302[15](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:15).json'
##[debug]GOOGLE_APPLICATION_CREDENTIALS='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-9007291eaec30[21](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:21)5.json'
##[debug]GOOGLE_GHA_CREDS_PATH='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-9007291eaec30215.json'
##[debug]CLOUDSDK_CORE_PROJECT='unstoppable-domains-staging'
##[debug]CLOUDSDK_PROJECT='unstoppable-domains-staging'
##[debug]GCLOUD_PROJECT='unstoppable-domains-staging'
##[debug]GCP_PROJECT='unstoppable-domains-staging'
##[debug]GOOGLE_CLOUD_PROJECT='unstoppable-domains-staging'
##[debug]Finishing: Login to Google Cloud
Then, after running google-github-actions/setup-gcloud@v0
, in two out of four identical runners, I get:
##[debug]Evaluating: steps.auth.outputs.credentials_file_path
##[debug]Evaluating Index:
##[debug]..Evaluating Index:
##[debug]....Evaluating Index:
##[debug]......Evaluating steps:
##[debug]......=> Object
##[debug]......Evaluating String:
##[debug]......=> 'auth'
##[debug]....=> Object
##[debug]....Evaluating String:
##[debug]....=> 'outputs'
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'credentials_file_path'
##[debug]=> '/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-f88746354[2](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:2)fb167a.json'
##[debug]Result: '/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-f88746[3](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:3)5[4](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:4)2fb167a.json'
##[debug]Evaluating condition for step: 'Run TypeORM migrations'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Run TypeORM migrations
##[debug]Loading inputs
##[debug]Loading env
Run yarn workspace backend db:migration:run
yarn workspace backend db:migration:run
shell: /usr/bin/bash -e {0}
env:
CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-f887463[5](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:5)42fb1[6](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:6)[7](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:7)a.json
GOOGLE_APPLICATION_CREDENTIALS: /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-f[8](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:8)87463542fb167a.json
GOOGLE_GHA_CREDS_PATH: /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-f887463542fb167a.json
CLOUDSDK_CORE_PROJECT: unstoppable-domains-staging
CLOUDSDK_PROJECT: unstoppable-domains-staging
GCLOUD_PROJECT: unstoppable-domains-staging
GCP_PROJECT: unstoppable-domains-staging
GOOGLE_CLOUD_PROJECT: unstoppable-domains-staging
APP_ENV: e2e
##[debug]/usr/bin/bash -e /home/runner/work/_temp/c[9](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:9)6e0527-df69-4e39-aaaa-fe29c93fc39f.sh
yarn workspace v1.22.18
yarn run v1.22.18
$ yarn typeorm migration:run
$ ts-node -r tsconfig-paths/register -T ./lib/cli.ts migration:run
Error during migration run:
Error: 16 UNAUTHENTICATED: Failed to retrieve auth metadata with error: Error code [object Object]
at Object.callErrorFromStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/call.ts:81:24)
at Object.onReceiveStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/client.ts:334:36)
at Object.onReceiveStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
at Object.onReceiveStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
at /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/call-stream.ts:276:24
at processTicksAndRejections (node:internal/process/task_queues:78:[11](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:11)) {
code: [16](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:16),
details: 'Failed to retrieve auth metadata with error: Error code [object Object]',
metadata: Metadata { internalRepr: Map(0) {}, options: {} },
note: 'Exception occurred in retry method that was not classified as transient'
}
error Command failed with exit code 1.
}
Hi @selfagency - that tells me that the authentication step is succeeding (you're getting an auth token), but then Cloud Storage is rejecting the token. If this is a recurring issue, I would recommend opening an issue with Google Cloud support. 503 is generally a server-side issue.
@sethvargo I opened an issue with Google Cloud Support and they told me that debugging Workflow Identity Federation issues is out of scope.
Hi @selfagency the issue is not Workload Identity Federation (we proved that when adding token_format: access_token
did not create an error. From the logs output, Cloud Storage is returning HTTP 503 responses, and unfortunately that's not something we can actually see or debug on this team.
Hi @selfagency - were you able to resolve this?
Unfortunately we're still getting random errors. Google Cloud support thinks it's GitHub's fault. GitHub's OIDC team is now investigating after I asked an executive for help.
Okay I'll leave this open for now. I do think the issue is outside of this library, since it's just calling the upstream APIs.
Mischa (GitHub Support) Apr 22, 2022, 4:06 PM UTC
Hello daniel,
Thank you for your patience while I check in with Engineering.
Our Engineering team relayed that since the google-github-actions/auth Action is able to successfully get a token, they do not suspect that the issue stems from the OIDC flow from the GitHub side. A 503 error from the upstream API indicates a problem with the service in question being temporarily unavailable, which we would not be able to dig into further on our end.
Regards,
Michael Lee Enterprise Support Engineer
That response makes sense. The logs clearly shows auth
is getting GitHub's OIDC token and it's exchanged for a GCP auth credential.
I'm not sure if this is related, but we're recently having authentication issues when pushing images, and we haven't touch a single thing on the GCP side https://github.com/ZcashFoundation/zebra/runs/6222308696?check_suite_focus=true#step:9:2120
Hi @selfagency
Is this still happening? Do you have any additional information?
TL;DR
We're experiencing ongoing random auth errors across Google services after authenticating with this action.
Expected behavior
We've had no issues with auth until today. We rolled out a new parallelized testing workflow earlier that's been working fine in testing up until it rolled out today and some of these auth errors are happening in non-parallelized jobs so I'm not sure it's related. Usually, this all works without a hitch, but Google Cloud's not showing any service outages on its status page.
Observed behavior
Action YAML
Log output
Additional information
No response