search

actions / upload-artifact

MIT License
3.26k stars 739 forks source link

[bug] uploads failing with "unable to verify the first certificate" #422

Closed risotto-master closed 11 months ago

risotto-master commented 1 year ago

What happened?

A workflow we've been using for some time now without issues has started flaring up this morning. We run a upload-artifact step to store testing suite reports.

Here's the chunk of the logs:

Run actions/upload-artifact@v3
  with:
    name: playwright-report
    path: ./playwright-report/
    retention-days: 1
    if-no-files-found: warn
With the provided path, there will be 215 files uploaded
Starting artifact upload
For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging
Artifact name is valid!
Container for artifact "playwright-report" successfully created. Starting upload of file(s)
An error has been caught http-client index 1, retrying the upload
Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
    at TLSSocket.emit (node:events:527:28)
    at TLSSocket._finishInit (node:_tls_wrap:946:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:727:12) {
  code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE'
}
Exponential backoff for retry #1. Waiting for 5009 milliseconds before continuing the upload at offset 0
An error has been caught http-client index 0, retrying the upload
Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
    at TLSSocket.emit (node:events:527:28)
    at TLSSocket._finishInit (node:_tls_wrap:946:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:727:12) {
  code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE'
}
Exponential backoff for retry #1. Waiting for 4877 milliseconds before continuing the upload at offset 0
Finished backoff for retry #1, continuing with upload
Finished backoff for retry #1, continuing with upload
An error has been caught http-client index 0, retrying the upload
Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
    at TLSSocket.emit (node:events:527:28)
    at TLSSocket._finishInit (node:_tls_wrap:946:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:727:12) {
  code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE'
}
Exponential backoff for retry #2. Waiting for 9661 milliseconds before continuing the upload at offset 0
An error has been caught http-client index 1, retrying the upload
Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
An error has been caught http-client index 0, retrying the upload
Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
    at TLSSocket.emit (node:events:527:28)
    at TLSSocket._finishInit (node:_tls_wrap:946:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:727:12) {
  code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE'
}
Exponential backoff for retry #1. Waiting for 4799 milliseconds before continuing the upload at offset 0
Finished backoff for retry #4, continuing with upload
An error has been caught http-client index 1, retrying the upload
Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
    at TLSSocket.emit (node:events:527:28)
    at TLSSocket._finishInit (node:_tls_wrap:946:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:727:12) {
  code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE'
}
Exponential backoff for retry #5. Waiting for 27972 milliseconds before continuing the upload at offset 0
Finished backoff for retry #1, continuing with upload
Total file count: 215 ---- Processed file #132 (61.3%)
An error has been caught http-client index 0, retrying the upload
Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
    at TLSSocket.emit (node:events:527:28)
    at TLSSocket._finishInit (node:_tls_wrap:946:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:727:12) {
  code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE'
}
Exponential backoff for retry #1. Waiting for 6504 milliseconds before continuing the upload at offset 0
Finished backoff for retry #1, continuing with upload
An error has been caught http-client index 0, retrying the upload
Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
    at TLSSocket.emit (node:events:527:28)
    at TLSSocket._finishInit (node:_tls_wrap:946:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:727:12) {
  code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE'
}
Exponential backoff for retry #1. Waiting for 6133 milliseconds before continuing the upload at offset 0
Total file count: 215 ---- Processed file #135 (62.7%)
Finished backoff for retry #1, continuing with upload
An error has been caught http-client index 0, retrying the upload
Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
    at TLSSocket.emit (node:events:527:28)
    at TLSSocket._finishInit (node:_tls_wrap:946:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:727:12) {
  code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE'
}
Exponential backoff for retry #2. Waiting for 11349 milliseconds before continuing the upload at offset 0
Total file count: 215 ---- Processed file #135 (62.7%)
Finished backoff for retry #2, continuing with upload
An error has been caught http-client index 0, retrying the upload
Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
    at TLSSocket.emit (node:events:527:28)
    at TLSSocket._finishInit (node:_tls_wrap:946:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:727:12) {
  code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE'
}
Exponential backoff for retry #3. Waiting for 18537 milliseconds before continuing the upload at offset 0
Finished backoff for retry #5, continuing with upload
An error has been caught http-client index 1, retrying the upload
Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
    at TLSSocket.emit (node:events:527:28)
    at TLSSocket._finishInit (node:_tls_wrap:946:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:727:12) {
  code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE'
}
Retry limit has been reached for chunk at offset 0 to https://pipelinesghubeus6.actions.githubusercontent.com/lYml6ozTlKSYZG8eLqKoXpuSIwnN1JX8iJfx1OQEG5L9B3aRsG/_apis/resources/Containers/9276249?itemPath=playwright-report%2Fdata%2Fb2a42e9c0355cf619a8c5aa6a5c344c6f6e8dc7f.webm
Warning: Aborting upload for /home/runner/work/myrepo/playwright-report/data/b2a42e9c0355cf619a8c5aa6a5c344c6f6e8dc7f.webm due to failure
Error: aborting artifact upload
Total file count: 215 ---- Processed file #136 (63.2%)
Total file count: 215 ---- Processed file #136 (63.2%)
Finished backoff for retry #3, continuing with upload
Total size of all the files uploaded is 45279621 bytes
File upload process has finished. Finalizing the artifact upload
Finalize artifact upload - Attempt 1 of 5 failed with error: unable to verify the first certificate
Finalize artifact upload - Attempt 2 of 5 failed with error: unable to verify the first certificate
Upload finished. There were 79 items that failed to upload

The raw size of all the files that were specified for upload is 46417991 bytes
The size of all the files that were uploaded is 45279621 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage

Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads

What did you expect to happen?

Expected upload to proceed like it usually does.

How can we reproduce it?

Not quite sure, it's a private repo. Again, I don't think anything we do is special or has changed to affect this.

Anything else we need to know?

We're running Ubuntu latest and are in beta for larger runners.

What version of the action are you using?

actions/upload-artifact@v3

What are your runner environments?

linux

Are you on GitHub Enterprise Server? If so, what version?

No response

aeoriwnd commented 1 year ago

Experiencing the same issue

MatanHeledPort commented 1 year ago

Also experiencing this. Looks more like a runner issue

adevick commented 1 year ago

same! for me

dystro-rb commented 1 year ago

Experiencing this issue as well.

NeoHsu commented 1 year ago

Also experiencing this. Looks more like a runner issue

Yes, I also believe that there could be an issue with certain GitHub Action runners. I utilized matrix concurrency in my workflow to execute the same job, and a few of the jobs were successful. Thankfully, after multiple attempts of triggering it, all of them eventually passed.

MatanHeledPort commented 1 year ago

Seems like this was fixed? Can anyone confirm?

tophercf commented 1 year ago

it seems to be fixed for our workflow runs, thanks to whoever fixed!

konradpabjan commented 11 months ago

This was fixed a while back (and it hasn't happened since). Closing

Jarod1662 commented 7 months ago

I've recently started seeing this on my workflow. any one else seeing the same thing? If so, how is this resolved?

BrendenWalker commented 7 months ago

I'm getting this in my workflow from 1 of 2 self-hosted windows runners using a dead simple workflow (see following).

Considering I can duplicate reliably on one self-hosted windows runner but not the other, this sounds like a local config issue OR the runner operating differently based on windows version.

For me the workflow is fine on Windows Server 2012R2, fails on Server 2016. On both I tried browsing to https://results-receiver.actions.githubusercontent.com/ and I get a 404 and no issues with the server certificate. Both systems have the same security tooling (ESET, Arctic Wolf, Wazuh).

I noticed issues connecting to tls1.2 on Server 2016. Fixed that, restarted actions runner and still having the issue so I don't think it's related to tls1.2

EDIT: I suspect this is related to some security controls in place. I tested on a clean/patched Server2016 VM and cannot duplicate there (after enabling tls 1.2)

Here's a section of debug log

2024-04-23T13:31:13.1541057Z ##[debug][Request] CreateArtifact https://results-receiver.actions.githubusercontent.com/twirp/github.actions.results.api.v1.ArtifactService/CreateArtifact
2024-04-23T13:31:13.2445942Z Attempt 1 of 5 failed with error: unable to verify the first certificate. Retrying request in 3000 ms...
2024-04-23T13:31:16.2783605Z Attempt 2 of 5 failed with error: unable to verify the first certificate. Retrying request in 6252 ms...
2024-04-23T13:31:22.5654757Z Attempt 3 of 5 failed with error: unable to verify the first certificate. Retrying request in 9856 ms...
2024-04-23T13:31:32.4540594Z Attempt 4 of 5 failed with error: unable to verify the first certificate. Retrying request in 14866 ms...
2024-04-23T13:31:47.3562686Z ##[error]Failed to CreateArtifact: Failed to make request after 5 attempts: unable to verify the first certificate
2024-04-23T13:31:47.3646246Z ##[debug]Node Action run completed with exit code 1
2024-04-23T13:31:47.3660652Z ##[debug]Finishing: Upload Artifacts

And my test workflow

name: testartifacts

on:
  workflow_dispatch:

jobs:
  upload-artifacts:
    runs-on: [self-hosted,clarion]
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Upload Artifacts
        uses: actions/upload-artifact@v4
        with:
          name: build-artifacts
          path: 
            Build/

  TestDownload:
    runs-on: [self-hosted,clarion]
    needs: upload-artifacts
    steps:
      - name: Download Artifacts
        uses: actions/download-artifact@v4
        with:
          name: build-artifacts

      - name: Verify download
        shell: powershell
        run: dir ./Build
GammaGames commented 1 month ago

If you need a temporary workaround (for self-signed certs or the like) you can set NODE_TLS_REJECT_UNAUTHORIZED:

  uses: actions/upload-artifact@v4
  with:
    name: build-artifacts
    path: Build/
  env:
    NODE_TLS_REJECT_UNAUTHORIZED: 0

A note from the docs:

This makes TLS, and HTTPS by extension, insecure. The use of this environment variable is strongly discouraged.