search

google-github-actions / upload-cloud-storage

A GitHub Action for uploading files to a Google Cloud Storage (GCS) bucket.
https://cloud.google.com/storage
Apache License 2.0
206 stars 49 forks source link

Error: google-github-actions/upload-cloud-storage failed with: invalid response body while trying to fetch https://sts.googleapis.com/v1/token: read ECONNRESET #320

Closed adaszko closed 1 year ago

adaszko commented 1 year ago

TL;DR

Hi 👋

I’m getting that error since yesterday (it worked fine before that): Error: google-github-actions/upload-cloud-storage failed with: invalid response body while trying to fetch https://sts.googleapis.com/v1/token: read ECONNRESET

Here’s how I call the action: https://github.com/radicle-dev/heartwood/blob/master/.github/workflows/release.yml#L115

Expected behavior

Uploads shoud still work

Observed behavior

Uploads fail

Action YAML

name: Release

on:
  push:
    branches: [master]

jobs:
  release-linux-binaries:
    permissions:
      contents: 'read'
      id-token: 'write'
    runs-on: ubuntu-20.04
    strategy:
      matrix:
        target: [x86_64-unknown-linux-musl, aarch64-unknown-linux-musl]
        include:
          - target: x86_64-unknown-linux-musl
            docker_image: registry.gitlab.com/rust_musl_docker/image:stable-latest
            docker_options: -v /home/runner/work/heartwood/heartwood:/workdir -v /home/runner/.cargo/git:/root/.cargo/git -v /home/runner/.cargo/registry:/root/.cargo/registry
          - target: aarch64-unknown-linux-musl
            docker_image: messense/rust-musl-cross:aarch64-musl
            docker_options: -v /home/runner/work/heartwood/heartwood:/home/rust/src -v /home/runner/.cargo/git:/root/.cargo/git -v /home/runner/.cargo/registry:/root/.cargo/registry
    steps:
      - name: Checkout source code
        uses: actions/checkout@v3
      - name: Configure build cache
        uses: actions/cache@v3
        with:
          path: |
            ~/.cargo/registry
            ~/.cargo/git
            target
          key: cargo-${{ matrix.target }}-${{ hashFiles('**/Cargo.lock') }}
      - name: Build the binaries
        uses: addnab/docker-run-action@v3
        with:
          image: ${{ matrix.docker_image }}
          options: ${{ matrix.docker_options }}
          run: |
            rustup target add ${{ matrix.target }}
            cargo build --target=${{ matrix.target }} --package=radicle-httpd --release
            cargo build --target=${{ matrix.target }} --package=radicle-node --release
            cargo build --target=${{ matrix.target }} --bin rad --release
            cargo build --target=${{ matrix.target }} --bin git-remote-rad --release
            chmod --recursive go+r /root/.cargo/registry
      - shell: bash
        run: |
          staging="${{ matrix.target }}"
          mkdir -p "$staging"
          cp target/${{ matrix.target }}/release/radicle-httpd "$staging"/
          cp target/${{ matrix.target }}/release/radicle-node "$staging"/
          cp target/${{ matrix.target }}/release/rad "$staging"/
          cp target/${{ matrix.target }}/release/git-remote-rad "$staging"/
          tar czf "$staging.tar.gz" "$staging"
          cp "$staging.tar.gz" "$staging"/
      - id: 'auth'
        uses: 'google-github-actions/auth@v1'
        with:
          workload_identity_provider: 'projects/281042598092/locations/global/workloadIdentityPools/github-actions/providers/google-cloud'
          service_account: 'github-actions@radicle-services.iam.gserviceaccount.com'
      - name: 'Set up Cloud SDK'
        uses: 'google-github-actions/setup-gcloud@v1'
      - id: 'upload-file'
        uses: 'google-github-actions/upload-cloud-storage@v1'
        with:
          path: ${{ matrix.target }}
          destination: heartwood-artifacts/${{ github.sha }}/

  release-macos-binaries:
    permissions:
      contents: 'read'
      id-token: 'write'
    runs-on: macos-11
    strategy:
      matrix:
        target:
          - aarch64-apple-darwin
          - x86_64-apple-darwin
    steps:
      - uses: actions/checkout@v3
      - name: Configure build cache
        uses: actions/cache@v3
        with:
          path: |
            ~/.cargo/registry
            ~/.cargo/git
            target
          key: cargo-${{ matrix.target }}-${{ hashFiles('**/Cargo.lock') }}
      - uses: dtolnay/rust-toolchain@1.67
      - name: Build the release binaries
        run: |
          rustup target add ${{ matrix.target }}
          cargo build --target ${{ matrix.target }} --package=radicle-httpd --release
          cargo build --target ${{ matrix.target }} --package=radicle-node --release
          cargo build --target ${{ matrix.target }} --bin rad --release
          cargo build --target ${{ matrix.target }} --bin git-remote-rad --release
      - shell: bash
        run: |
          staging="${{ matrix.target }}"
          mkdir -p "$staging"
          cp target/${{ matrix.target }}/release/radicle-httpd "$staging"/
          cp target/${{ matrix.target }}/release/radicle-node "$staging"/
          cp target/${{ matrix.target }}/release/rad "$staging"/
          cp target/${{ matrix.target }}/release/git-remote-rad "$staging"/
          tar czf "$staging.tar.gz" "$staging"
          cp "$staging.tar.gz" "$staging"/
      - id: 'auth'
        uses: 'google-github-actions/auth@v1'
        with:
          workload_identity_provider: 'projects/281042598092/locations/global/workloadIdentityPools/github-actions/providers/google-cloud'
          service_account: 'github-actions@radicle-services.iam.gserviceaccount.com'
      - name: 'Set up Cloud SDK'
        uses: 'google-github-actions/setup-gcloud@v1'
      - id: 'upload-file'
        uses: 'google-github-actions/upload-cloud-storage@v1'
        with:
          path: ${{ matrix.target }}
          destination: heartwood-artifacts/${{ github.sha }}

Log output

Run google-github-actions/upload-cloud-storage@v1
Warning: The "process_gcloudignore" option is true, but no .gcloudignore file was found. If you do not intend to process a gcloudignore file, set "process_gcloudignore" to false.
Upload files
  Uploading /Users/runner/work/heartwood/heartwood/x86_64-apple-darwin/x86_64-apple-darwin.tar.gz to gs://heartwood-artifacts/0b7f169c8bcd51a5e01a83ec2a00fe5aea31207c/x86_64-apple-darwin/x86_64-apple-darwin.tar.gz
  Uploading /Users/runner/work/heartwood/heartwood/x86_64-apple-darwin/radicle-node to gs://heartwood-artifacts/0b7f169c8bcd51a5e01a83ec2a00fe5aea31207c/x86_64-apple-darwin/radicle-node
  Uploading /Users/runner/work/heartwood/heartwood/x86_64-apple-darwin/radicle-httpd to gs://heartwood-artifacts/0b7f169c8bcd51a5e01a83ec2a00fe5aea31207c/x86_64-apple-darwin/radicle-httpd
  Uploading /Users/runner/work/heartwood/heartwood/x86_64-apple-darwin/rad to gs://heartwood-artifacts/0b7f169c8bcd51a5e01a83ec2a00fe5aea31207c/x86_64-apple-darwin/rad
  Uploading /Users/runner/work/heartwood/heartwood/x86_64-apple-darwin/git-remote-rad to gs://heartwood-artifacts/0b7f169c8bcd51a5e01a83ec2a00fe5aea31207c/x86_64-apple-darwin/git-remote-rad
  Error: google-github-actions/upload-cloud-storage failed with: invalid response body while trying to fetch https://sts.googleapis.com/v1/token: read ECONNRESET

  node:internal/process/promises:279
              triggerUncaughtException(err, true /* fromPromise */);
              ^
  FetchError: Invalid response body while trying to fetch https://sts.googleapis.com/v1/token: read ECONNRESET
      at Gunzip.<anonymous> (/Users/runner/work/_actions/google-github-actions/upload-cloud-storage/v1/dist/index.js:216:44699)
      at Gunzip.emit (node:events:539:35)
      at emitErrorNT (node:internal/streams/destroy:157:8)
      at emitErrorCloseNT (node:internal/streams/destroy:122:3)
      at processTicksAndRejections (node:internal/process/task_queues:83:21) {
    type: 'system',
    errno: 'ECONNRESET',
    code: 'ECONNRESET',
    config: {
      url: 'https://sts.googleapis.com/v1/token',
      method: 'POST',
      headers: {
        'Content-Type': 'application/x-www-form-urlencoded',
        'User-Agent': 'google-api-nodejs-client/8.7.0',
        'x-goog-api-client': 'gl-node/16.16.0 auth/8.7.0',
        Accept: 'application/json'
      },
      data: 'grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Atoken-exchange&audience=%2F%2Fiam.googleapis.com%2Fprojects%2F28104[25](https://github.com/radicle-dev/heartwood/actions/runs/4576655428/jobs/8093391678#step:9:26)98092%2Flocations%2Fglobal%2FworkloadIdentityPools%2Fgithub-actions%2Fproviders%2Fgoogle-cloud&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform&requested_token_type=urn%3Aietf%3Aparams%3Aoauth%3Atoken-type%3Aaccess_token&subject_token=***',
      responseType: 'json',
      paramsSerializer: [Function: paramsSerializer],
      body: 'grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Atoken-exchange&audience=%2F%2Fiam.googleapis.com%2Fprojects%2F[28](https://github.com/radicle-dev/heartwood/actions/runs/4576655428/jobs/8093391678#step:9:29)10[42](https://github.com/radicle-dev/heartwood/actions/runs/4576655428/jobs/8093391678#step:9:43)598092%2Flocations%2Fglobal%2FworkloadIdentityPools%2Fgithub-actions%2Fproviders%2Fgoogle-cloud&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform&requested_token_type=urn%3Aietf%3Aparams%3Aoauth%3Atoken-type%3Aaccess_token&subject_token=***',
      validateStatus: [Function: validateStatus]
    }
  }

Additional information

No response

sethvargo commented 1 year ago

Hi @adaszko

ECONNRESET means the TCP connection between the runner and Google Cloud was reset. This usually indicates a faulty network connection or a transient issue. Is this still occurring?

This error happens well beyond our realm of control.

adaszko commented 1 year ago

Hi. Yes, still happening. Latest instance from half an hour ago (not sure if you have permission to see that): https://github.com/radicle-dev/heartwood/actions/runs/4597679331/jobs/8120701331

sethvargo commented 1 year ago

@bcoe it looks like the GCS client library is reading a new token from the sts endpoint on each file that's uploaded - is that correct?

nikiizvorski commented 1 year ago

We are having the same problem. Could you help us out to solve it since we are blocked in production?

This is what we use currently: uses: google-github-actions/upload-cloud-storage@main

I have sorted it out by downgrading to the google-github-actions/upload-cloud-storage@v0 and i tested google-github-actions/upload-cloud-storage@v1 which failed like main.

CB-GuangyaoXie commented 1 year ago

Seeing the same issue