search

google-github-actions / auth

A GitHub Action for authenticating to Google Cloud.
https://cloud.google.com/iam
Apache License 2.0
953 stars 195 forks source link

Inconsistent but persistent auth errors starting today #160

Closed selfagency closed 2 years ago

selfagency commented 2 years ago

TL;DR

We're experiencing ongoing random auth errors across Google services after authenticating with this action.

Expected behavior

We've had no issues with auth until today. We rolled out a new parallelized testing workflow earlier that's been working fine in testing up until it rolled out today and some of these auth errors are happening in non-parallelized jobs so I'm not sure it's related. Usually, this all works without a hitch, but Google Cloud's not showing any service outages on its status page.

Observed behavior

google-github-actions/setup-gcloud failed with: failed to execute command `gcloud --quiet auth login --cred-file /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-a1b71b0c5598d105.json`: ERROR: gcloud crashed (OAuthError): ("Error code {'code': 503, 'message': 'The service is currently unavailable.', 'status': 'UNAVAILABLE'}", '{\n  "error": {\n    "code": 503,\n    "message": "The service is currently unavailable.",\n    "status": "UNAVAILABLE"\n  }\n}\n')
Error: 16 UNAUTHENTICATED: Failed to retrieve auth metadata with error: Error code [object Object]
    at Object.callErrorFromStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/call.ts:81:24)
    at Object.onReceiveStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/client.ts:334:36)
    at Object.onReceiveStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
    at Object.onReceiveStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:[48](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5824013165?check_suite_focus=true#step:15:48))
    at /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/call-stream.ts:276:24
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  code: 16,
  details: 'Failed to retrieve auth metadata with error: Error code [object Object]',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} },
  note: 'Exception occurred in retry method that was not classified as transient'
}
Error: google-github-actions/upload-cloud-storage failed with error code [object Object]

Action YAML

name: E2E tests
on:
  schedule:
    - cron: '0 0 * * *'
  push:
    branches:
      - master
      - staging
  pull_request:
    types:
      - opened
      - reopened
      - synchronize
  workflow_dispatch:
  # workflow_run:
  #   workflows: ['Build']
  #   types: ['completed']
concurrency:
  group: e2e_${{ github.ref }}_${{ github.event_name }}
  cancel-in-progress: true
env:
  GCP_PROJECT: unstoppable-domains-staging
  PACKAGE: e2e
  BAIL: false
jobs:
  setup:
    name: Setup workflow run
    uses: './.github/workflows/_setup.yml'
    with:
      trigger: ${{ github.event_name }}
      actor: ${{ github.actor }}
      title: ${{ github.event.pull_request.title }}

  pre-cache:
    name: Pre-cache frontend build
    needs: ['setup']
    if: ${{ needs.setup.outputs.skipped == 'true' || (needs.setup.outputs.e2e == 'true' && needs.setup.outputs.hotfix == 'false') }}
    uses: './.github/workflows/_precache.yml'
    with:
      environment: e2e

  test:
    name: End-to-End tests
    runs-on: ubuntu-latest
    needs: ['setup', 'pre-cache']
    if: ${{ needs.setup.outputs.skipped == 'true' || (needs.setup.outputs.e2e == 'true' && needs.setup.outputs.hotfix == 'false') }}
    outputs:
      failed: ${{ steps.tests.outcome != 'success' }}
      outcome: ${{ steps.tests.outcome }}
      started: ${{ steps.start_timer.outputs.started }}
      finished: ${{ steps.end_timer.outputs.finished }}
      duration: ${{ steps.end_timer.outputs.duration }}
      build: ${{ needs.pre-cache.outputs.cache-hit != 'true' }}
    strategy:
      fail-fast: false
      matrix:
        ci_node_total: [4]
        ci_node_index: [0, 1, 2, 3]
    permissions:
      id-token: write
      contents: read
      pull-requests: write
      issues: write
    services:
      postgres:
        image: 'postgres:11.6'
        ports: ['5432:5432']
        options: '--health-cmd pg_isready --health-interval 10s --health-timeout 5s
          --health-retries 5'
        env:
          POSTGRES_DB: unstoppable_website_test
      redis:
        image: 'redis:5.0'
        ports:
          - '6379:6379'
        options: '--entrypoint redis-server'
    steps:
      - name: Checkout repo
        uses: 'actions/checkout@v2'

      - name: Configure Node.js ${{ secrets.NODE_VERSION }}
        uses: 'actions/setup-node@v2'
        with:
          node-version: ${{ secrets.NODE_VERSION }}
          cache: yarn

      - name: Retrieve project Yarn cache
        id: cache
        uses: 'actions/cache@v2'
        with:
          path: |
            ./node_modules
            **/node_modules
            !ci/node_modules
          key: ${{ runner.os }}-dev-${{ hashFiles('./yarn.lock') }}

      - name: Install dependencies
        if: ${{ steps.cache.outputs.cache-hit != 'true' }}
        run: yarn install --prefer-offline --frozen-lockfile 2> >(grep -v warning 1>&2)

      - name: Retrieve CI Yarn cache
        id: ci-cache
        uses: 'actions/cache@v2'
        with:
          path: ./ci/node_modules
          key: ${{ runner.os }}-ci-${{ hashFiles('./ci/yarn.lock') }}

      - name: Install CI dependencies
        if: ${{ steps.ci-cache.outputs.cache-hit != 'true' }}
        working-directory: ./ci
        run: yarn install --prefer-offline --frozen-lockfile 2> >(grep -v warning 1>&2)

      - name: Retrieve shared library build cache
        uses: 'actions/cache@v2'
        id: shared-cache
        with:
          path: shared/dist
          key: ${{ runner.os }}-shared-${{ hashFiles('./yarn.lock', './shared/src/**/*.ts', './shared/src/**/*.js', './shared/**/*.json') }}

      - name: Retrieve frontend build cache
        uses: 'actions/cache@v2'
        id: frontend-cache
        with:
          path: |
            client/build
            client/.next
            ~/.next
          key: ${{ runner.os }}-frontend-e2e-${{ hashFiles('./yarn.lock', './client/**/*.ts', './client/**/*.tsx', './client/**/*.js', './client/**/*.jsx', './client/**/*.json') }}

      - name: Retrieve Cypress cache
        uses: 'actions/cache@v2'
        id: cypress_cache
        with:
          path: ~/.cache/Cypress
          key: ${{ runner.os }}-cache-cypress-9.5.3

      - name: Install Cypress
        if: ${{ steps.cypress_cache.outputs.cache-hit != 'true' }}
        working-directory: ./e2e
        run: yarn cypress install

      - name: Login to Google Cloud
        id: auth
        uses: 'google-github-actions/auth@v0'
        with:
          workload_identity_provider: ${{ secrets.WORKLOAD_ID_PROVIDER_STAGING }}
          service_account: ${{ secrets.SERVICE_ACCOUNT_STAGING }}

      - name: Configure Google Cloud
        uses: 'google-github-actions/setup-gcloud@v0'

      - name: Run TypeORM migrations
        env:
          APP_ENV: e2e
          GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.auth.outputs.credentials_file_path }}
        run: 'yarn workspace backend db:migration:run'

      - name: Generate UUID
        id: uuid
        run: echo "::set-output name=uuid::$(uuidgen)"

      - name: Run E2E tests
        id: tests
        uses: 'cypress-io/github-action@v3.0.3'
        continue-on-error: true
        env:
          NODE_ENV: development
          APP_ENV: e2e
          API_BASE_URL: 'http://localhost:8080'
          CLIENT_URL: 'http://localhost:3000'
          CYPRESS_PROJECT_ID: 9g89i4
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
          RECORD: ${{ github.event_name == 'schedule' }}
          KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS: ${{ secrets.KNAPSACK_E2E }}
          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
          KNAPSACK_PRO_TEST_FILE_PATTERN: '${{ github.workspace }}/e2e/cypress/integration/*.ts'
          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
        with:
          command: '${{ github.workspace }}/ci/node_modules/.bin/knapsack-pro-cypress --configFile ${{ github.workspace }}/e2e/cypress.ci.json --headless --reporter ${{ github.workspace }}/ci/node_modules/mochawesome --reporter-options html=false,json=true,reportDir=${{ github.workspace }}/e2e,reportFilename=test-report-${{ steps.uuid.outputs.uuid }},overwrite=false --browser chrome --spec ${{ github.workspace }}/e2e/cypress/integration/**/*.ts --record=${{ env.RECORD }}'
          install: false
          start: 'yarn workspace backend start:e2e, yarn workspace client start:e2e'
          wait-on: 'http://localhost:8080/api/alive, http://localhost:3000'
          working-directory: ${{ github.workspace }}/e2e

      - name: Upload test results to GitHub
        uses: actions/upload-artifact@v2
        continue-on-error: true
        with:
          name: ${{ env.PACKAGE }}_${{ github.sha }}
          path: |
            ${{ env.PACKAGE }}/test-report*
          retention-days: 1

  results:
    name: Validate and publish test results
    uses: './.github/workflows/_results.yml'
    needs: ['test', 'setup']
    if: ${{ needs.setup.outputs.skipped == 'true' || (needs.setup.outputs.e2e == 'true' && needs.setup.outputs.hotfix == 'false') }}
    with:
      package: e2e
      failed: ${{ needs.test.outputs.failed }}
      bailed: false
      install: ${{ needs.setup.outputs.cache-hit != 'true' }}
      build: ${{ needs.test.outputs.build }}
      sha: ${{ github.sha }}
      started: ${{ needs.setup.outputs.started }}
      outcome: ${{ needs.test.outputs.outcome }}
      report: 'https://storage.googleapis.com/unstoppable-devops/ci-artifacts/${{ github.run_id }}/e2e/test-report.html'
    secrets:
      WORKLOAD_ID_PROVIDER_STAGING: ${{ secrets.WORKLOAD_ID_PROVIDER_STAGING }}
      SERVICE_ACCOUNT_STAGING: ${{ secrets.SERVICE_ACCOUNT_STAGING }}
      NODE_VERSION: ${{ secrets.NODE_VERSION }}
      SONAR_TOKEN: ${{ secrets.SONARCLOUD_TOKEN }}
      CC_TEST_REPORTER_ID: ${{ secrets.CC_TEST_REPORTER_ID }}

Log output

##[debug]Evaluating condition for step: 'Login to Google Cloud'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Login to Google Cloud
##[debug]Register post job cleanup for action: google-github-actions/auth@v0
##[debug]Loading inputs
##[debug]Evaluating: secrets.WORKLOAD_ID_PROVIDER_STAGING
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'WORKLOAD_ID_PROVIDER_STAGING'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Evaluating: secrets.SERVICE_ACCOUNT_STAGING
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'SERVICE_ACCOUNT_STAGING'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Loading env
Run google-github-actions/auth@v0
##[debug]Using workload identity provider "***"
##[debug]ID token url is https://pipelines.actions.githubusercontent.com/JIdNBcQ6kMWGQ1Emvk4eWkVClMzQPdc1GeXypChXNSBsqbvPVO/00000000-0000-0000-0000-000000000000/_apis/distributedtask/hubs/Actions/plans/99435d5c-744f-40ef-8ace-52f2f419aeed/jobs/f72e1371-574f-5793-8b43-f4df44fd5814/idtoken?api-version=2.0&audience=https%3A%2F%2Fiam.googleapis.com%2F***
::add-mask::***
##[debug]Creating credentials file
Created credentials file at "/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-a1b71b0c5598d105.json"

::set-output name=credentials_file_path::/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-a1b71b0c5598d105.json
##[debug]steps.auth.outputs.credentials_file_path='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-a1b71b0c5598d105.json'

::set-output name=project_id::unstoppable-domains-staging
##[debug]steps.auth.outputs.project_id='unstoppable-domains-staging'
##[debug]Node Action run completed with exit code 0
##[debug]CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-a1b71b0c5598d105.json'
##[debug]GOOGLE_APPLICATION_CREDENTIALS='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-a1b71b0c5598d105.json'
##[debug]GOOGLE_GHA_CREDS_PATH='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-a1b71b0c5598d105.json'
##[debug]CLOUDSDK_PROJECT='unstoppable-domains-staging'
##[debug]CLOUDSDK_CORE_PROJECT='unstoppable-domains-staging'
##[debug]GCP_PROJECT='unstoppable-domains-staging'
##[debug]GCLOUD_PROJECT='unstoppable-domains-staging'
##[debug]GOOGLE_CLOUD_PROJECT='unstoppable-domains-staging'
##[debug]Finishing: Login to Google Cloud

Additional information

No response

selfagency commented 2 years ago

@sethvargo We're totally blocked here so it would be greatly appreciated if you could find time this morning to help me out. Thanks.

sethvargo commented 2 years ago

Hi @selfagency

Can you try adding "token_format: access_token" to the auth step temporarily? If the Action proceeds beyond the auth step, it means there's an issue with the upstream service. However, if the Action fails at the auth step (after adding that token), it means the issue is in the authentication handshake, and the specific error message will give us more information.

selfagency commented 2 years ago

@sethvargo 

##[debug]Evaluating condition for step: 'Login to Google Cloud'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Login to Google Cloud
##[debug]Register post job cleanup for action: google-github-actions/auth@v0
##[debug]Loading inputs
##[debug]Evaluating: secrets.WORKLOAD_ID_PROVIDER_STAGING
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'WORKLOAD_ID_PROVIDER_STAGING'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Evaluating: secrets.SERVICE_ACCOUNT_STAGING
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'SERVICE_ACCOUNT_STAGING'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Loading env
Run google-github-actions/auth@v0
##[debug]Using workload identity provider "***"
##[debug]ID token url is https://pipelines.actions.githubusercontent.com/JIdNBcQ6kMWGQ1Emvk4eWkVClMzQPdc1GeXypChXNSBsqbvPVO/00000000-0000-0000-0000-000000000000/_apis/distributedtask/hubs/Actions/plans/bedb98a3-f043-4f9e-88[2](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:2)7-c75a2c08e90b/jobs/a9[3](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:3)157a9-a0f6-5af8-a70b-8506ae[4](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:4)7[5](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:5)1b2/idtoken?api-version=2.0&audience=https%3A%2F%2Fiam.googleapis.com%2F***
::add-mask::***
##[debug]Creating credentials file
Created credentials file at "/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-9007291eaec30215.json"

::set-output name=credentials_file_path::/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-9007291eaec30215.json
##[debug]steps.auth.outputs.credentials_file_path='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-9007291eaec30215.json'

::set-output name=project_id::unstoppable-domains-staging
##[debug]steps.auth.outputs.project_id='unstoppable-domains-staging'
Warning: Overwriting existing environment variable GCP_PROJECT (was: "unstoppable-domains-staging")
##[debug]Creating access token
::add-mask::***

::set-output name=access_token::***
##[debug]steps.auth.outputs.access_token='***'

::set-output name=access_token_expiration::2022-04-05T18:4[6](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:6):56Z
##[debug]steps.auth.outputs.access_token_expiration='2022-04-05T18:46:56Z'
##[debug]Node Action run completed with exit code 0
##[debug]CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-900[7](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:7)2[9](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:9)1eaec302[15](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:15).json'
##[debug]GOOGLE_APPLICATION_CREDENTIALS='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-9007291eaec30[21](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:21)5.json'
##[debug]GOOGLE_GHA_CREDS_PATH='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-9007291eaec30215.json'
##[debug]CLOUDSDK_CORE_PROJECT='unstoppable-domains-staging'
##[debug]CLOUDSDK_PROJECT='unstoppable-domains-staging'
##[debug]GCLOUD_PROJECT='unstoppable-domains-staging'
##[debug]GCP_PROJECT='unstoppable-domains-staging'
##[debug]GOOGLE_CLOUD_PROJECT='unstoppable-domains-staging'
##[debug]Finishing: Login to Google Cloud

Then, after running google-github-actions/setup-gcloud@v0, in two out of four identical runners, I get:

##[debug]Evaluating: steps.auth.outputs.credentials_file_path
##[debug]Evaluating Index:
##[debug]..Evaluating Index:
##[debug]....Evaluating Index:
##[debug]......Evaluating steps:
##[debug]......=> Object
##[debug]......Evaluating String:
##[debug]......=> 'auth'
##[debug]....=> Object
##[debug]....Evaluating String:
##[debug]....=> 'outputs'
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'credentials_file_path'
##[debug]=> '/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-f88746354[2](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:2)fb167a.json'
##[debug]Result: '/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-f88746[3](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:3)5[4](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:4)2fb167a.json'
##[debug]Evaluating condition for step: 'Run TypeORM migrations'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Run TypeORM migrations
##[debug]Loading inputs
##[debug]Loading env
Run yarn workspace backend db:migration:run
  yarn workspace backend db:migration:run
  shell: /usr/bin/bash -e {0}
  env:
    CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-f887463[5](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:5)42fb1[6](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:6)[7](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:7)a.json
    GOOGLE_APPLICATION_CREDENTIALS: /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-f[8](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:8)87463542fb167a.json
    GOOGLE_GHA_CREDS_PATH: /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-f887463542fb167a.json
    CLOUDSDK_CORE_PROJECT: unstoppable-domains-staging
    CLOUDSDK_PROJECT: unstoppable-domains-staging
    GCLOUD_PROJECT: unstoppable-domains-staging
    GCP_PROJECT: unstoppable-domains-staging
    GOOGLE_CLOUD_PROJECT: unstoppable-domains-staging
    APP_ENV: e2e
##[debug]/usr/bin/bash -e /home/runner/work/_temp/c[9](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:9)6e0527-df69-4e39-aaaa-fe29c93fc39f.sh
yarn workspace v1.22.18
yarn run v1.22.18
$ yarn typeorm migration:run
$ ts-node -r tsconfig-paths/register -T ./lib/cli.ts migration:run
Error during migration run:
Error: 16 UNAUTHENTICATED: Failed to retrieve auth metadata with error: Error code [object Object]
    at Object.callErrorFromStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/call.ts:81:24)
    at Object.onReceiveStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/client.ts:334:36)
    at Object.onReceiveStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
    at Object.onReceiveStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
    at /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/call-stream.ts:276:24
    at processTicksAndRejections (node:internal/process/task_queues:78:[11](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:11)) {
  code: [16](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:16),
  details: 'Failed to retrieve auth metadata with error: Error code [object Object]',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} },
  note: 'Exception occurred in retry method that was not classified as transient'
}
error Command failed with exit code 1.
}
sethvargo commented 2 years ago

Hi @selfagency - that tells me that the authentication step is succeeding (you're getting an auth token), but then Cloud Storage is rejecting the token. If this is a recurring issue, I would recommend opening an issue with Google Cloud support. 503 is generally a server-side issue.

selfagency commented 2 years ago

@sethvargo I opened an issue with Google Cloud Support and they told me that debugging Workflow Identity Federation issues is out of scope.

sethvargo commented 2 years ago

Hi @selfagency the issue is not Workload Identity Federation (we proved that when adding token_format: access_token did not create an error. From the logs output, Cloud Storage is returning HTTP 503 responses, and unfortunately that's not something we can actually see or debug on this team.

sethvargo commented 2 years ago

Hi @selfagency - were you able to resolve this?

selfagency commented 2 years ago

Unfortunately we're still getting random errors. Google Cloud support thinks it's GitHub's fault. GitHub's OIDC team is now investigating after I asked an executive for help.

sethvargo commented 2 years ago

Okay I'll leave this open for now. I do think the issue is outside of this library, since it's just calling the upstream APIs.

selfagency commented 2 years ago

Mischa (GitHub Support) Apr 22, 2022, 4:06 PM UTC

Hello daniel,

Thank you for your patience while I check in with Engineering.

Our Engineering team relayed that since the google-github-actions/auth Action is able to successfully get a token, they do not suspect that the issue stems from the OIDC flow from the GitHub side. A 503 error from the upstream API indicates a problem with the service in question being temporarily unavailable, which we would not be able to dig into further on our end.

Regards,

Michael Lee Enterprise Support Engineer

sethvargo commented 2 years ago

That response makes sense. The logs clearly shows auth is getting GitHub's OIDC token and it's exchanged for a GCP auth credential.

gustavovalverde commented 2 years ago

I'm not sure if this is related, but we're recently having authentication issues when pushing images, and we haven't touch a single thing on the GCP side https://github.com/ZcashFoundation/zebra/runs/6222308696?check_suite_focus=true#step:9:2120

sethvargo commented 2 years ago

Hi @selfagency

Is this still happening? Do you have any additional information?