Upload fails with: triggerUncaughtException(err, true /* fromPromise */);

[cwolf24]

TL;DR

Hello,

unfortunately i sometimes get an uncaught exception while uploading a testreport (~ 500 files) See log output for exception details. As i see in the log, that several files are uploaded before the exception appears. It happens from time to time. Maybe 1/10 tries.

Do you know if it is possible to enable a retry or handle the exception within your package? That would be awesome to have this Job always pass. 🙂

Expected behavior

uploading files always works

Observed behavior

upload is not successful due to the exception

Action YAML

upload-report:
    name: Upload Report
    if: (success() || failure()) && needs.test-integration.result != 'skipped'
    runs-on: ubuntu-latest
    environment:
      name: dev-cluster
    env:
      JOB_NAME: ${{github.job}}
    needs:
      - test-integration
    steps:
      - name: Archive Report Download
        uses: actions/download-artifact@v3
        with:
          path: tests/output/allure-report
      - name: Authenticate to GCP
        uses: google-github-actions/auth@v1.0.0
        with:
          token_format: access_token
          workload_identity_provider: ${{ secrets.*** }}
          service_account: ${{ secrets.***}}
      - name: Authenticate to GKE cluster
        uses: google-github-actions/get-gke-credentials@v1.0.1
        with:
          cluster_name: ${{ secrets.*** }}
          location: ${{ secrets.*** }}
          use_internal_ip: true
      - name: Upload Report
        uses: google-github-actions/upload-cloud-storage@v1
        with:
          path: tests/output/allure-report
          parent: false
          destination: ${{ env.BUCKET_NAME }}/${{ env.BUCKET_PATH }}
          concurrency: 50

Log output

Run google-github-actions/upload-cloud-storage@v1
Upload files
Uploading /tests/output/allure-report/test-integration/index.html to gs://***
...
Uploading /tests/output/allure-report/test-integration/history/retry-trend.json
Uploading /tests/output/allure-report/test-integration/history/history.json to gs://***
Uploading /tests/output/allure-report/test-integration/history/history-trend.json to gs://***
Uploading /tests/output/allure-report/test-integration/history/duration-trend.json to gs://***

 node:internal/process/promises:279
              triggerUncaughtException(err, true /* fromPromise */);
              ^
  FetchError: request to https://pipelines.actions.githubusercontent.com/... failed, reason: 
connect ETIMEDOUT 13.107.42.16:443
      at ClientRequest.<anonymous> (/home/runner/work/_actions/google-github-actions/upload-cloud-storage/v1/dist/index.js:201:57777)
      at ClientRequest.emit (node:events:527:28)
      at TLSSocket.socketErrorListener (node:_http_client:454:9)
      at TLSSocket.emit (node:events:527:28)
      at emitErrorNT (node:internal/streams/destroy:157:8)
      at emitErrorCloseNT (node:internal/streams/destroy:122:3)
      at processTicksAndRejections (node:internal/process/task_queues:83:21) {

[sethvargo]

Hi @cwolf24 - we already do retries, so these would be failures after retries. Is it always occurring on the same file?

[cwolf24]

@sethvargo Unfortunately it's hard to say if it is related to a specific file, because the files (allure testreport) are created dynamically beforehand... and as far as i can see it happens - when it happens - with one of the *.json files from a subfolder (allure-report/test-integration/data/test-cases/939ed1e9943e7b51.json always after 2:11 minutes... i just created an allure report locally and it doesn't look like there are big files or broken files or anything.

Can i somehow enable debugging output or something else that may help analysing this issue?

[cwolf24]

@sethvargo I just did re-run a failed upload reports job and now it failed after a few seconds with:

 Error: google-github-actions/upload-cloud-storage failed with: retry limit exceeded -
  node:internal/process/promises:279
              triggerUncaughtException(err, true /* fromPromise */);
              ^

  Error: Retry limit exceeded - 
      at Upload.attemptDelayedRetry (/home/runner/work/_actions/google-github-actions/upload-cloud-storage/v1/dist/index.js:139:10919)
      at Upload.onResponse (/home/runner/work/_actions/google-github-actions/upload-cloud-storage/v1/dist/index.js:139:10345)
      at Upload.makeRequestStream (/home/runner/work/_actions/google-github-actions/upload-cloud-storage/v1/dist/index.js:139:10142)
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Upload.startUploading (/home/runner/work/_actions/google-github-actions/upload-cloud-storage/v1/dist/index.js:139:7703)

Is this maybe related?

[sethvargo]

Hi there - here are instructions for enabling the debug output for the complete GitHub Actions workflow run.

Is this using GitHub Actions managed runners or self-hosted?

What is the total size of the upload? I think GitHub imposes bandwidth limits (although I cannot find any supporting documentation).

[cwolf24]

Hi @sethvargo ,

thanks for the info, i now could optimize our upload by splitting it for single testruns, so that not all files will be uploaded at the same time. So this issue can be closed. Thanks again for your support!