[bug] Yet another slow/broken artifacts upload

[ab-spingo]

What happened?

Deployment process hangs at upload. We're using v2 for Windows (Azure AppService) environment. It worked fine 3 days before. Normally upload taked about 30 seconds, this time we had to cancel the run after 1,5 hour. Nothing has changed in files count or code structure.

Container for artifact ".net-app" successfully created. Starting upload of file(s)
Total file count: 568 ---- Processed file #2 (0.3%)
Total file count: 568 ---- Processed file #2 (0.3%)
Total file count: 568 ---- Processed file #2 (0.3%)
Total file count: 568 ---- Processed file #2 (0.3%)
Total file count: 568 ---- Processed file #2 (0.3%)
Total file count: 568 ---- Processed file #2 (0.3%)
Total file count: 568 ---- Processed file #2 (0.3%)
Total file count: 568 ---- Processed file #2 (0.3%)
Total file count: 568 ---- Processed file #2 (0.3%)
Total file count: 568 ---- Processed file #2 (0.3%)
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #1. Waiting for 5467 milliseconds before continuing the upload at offset 0
...
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #1. Waiting for 4549 milliseconds before continuing the upload at offset 0
Finished backoff for retry #1, continuing with upload
Total file count: 568 ---- Processed file #351 (61.7%)
Total file count: 568 ---- Processed file #351 (61.7%)
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #2. Waiting for 9089 milliseconds before continuing the upload at offset 0

We also got a couple of these in the log:

An error has been caught http-client index 1, retrying the upload
Error: connect ECONNREFUSED 20.253.95.3:443
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1278:16) {
  errno: -4078,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '20.253.95.3',
  port: 443
}

What did you expect to happen?

We should either get a way faster deployment or a clear message that there is a problem with target server.

Container for artifact ".net-app" successfully created. Starting upload of file(s)
Total file count: 568 ---- Processed file #[16](https://github.com/ItFaktoria/spingo-dev/actions/runs/5902357165/job/16010240459#step:6:17)4 (28.8%)
Total file count: 568 ---- Processed file #390 (68.6%)
Total size of all the files uploaded is 25860935 bytes
File upload process has finished. Finalizing the artifact upload
Artifact has been finalized. All files have been successfully uploaded!

How can we reproduce it?

This is the first time we have encountered that issue, not sure how to reproduce it - but from the issue log I see that others have had this problem before.

Anything else we need to know?

No response

What version of the action are you using?

v2

What are your runner environments?

window

Are you on GitHub Enterprise Server? If so, what version?

No response

[Chimpaya]

Yep, we are also having this problem

[chernyadev]

Also observing massive slowdown today

[chernyadev]

Seems to be ok by now: https://www.githubstatus.com/incidents/q8swpy90g6pp

[yauhen-vastraknutau-epam]

Same slowdown for us for a couple of days already.

[LeviPesin]

Still observing this slowdown that breaks jobs: https://github.com/warriors-life/yokohama-proxy/actions/runs/5978170788/job/16219709767

[Alan-Jowett]

+1

https://github.com/microsoft/ebpf-for-windows/actions/runs/6239448484/job/16938150347

Uploading a few GB of profile tracing as an artifact started failing with 503.

[LeviPesin]

In my case the problem was with an outdated version of the harden-runner.

[rectified95]

@konradpabjan please reach out - MS projects are hitting this.

[konradpabjan]

V4 upload-artifact and download-artifact have officially dropped today! See https://github.blog/changelog/2023-12-14-github-actions-artifacts-v4-is-now-generally-available/

They're effectively total rewrites from scratch that improve on every aspect. v4 is major re-architecture that we've been working on for well over a year. Upload and downloads are orders of magnitudes faster and more reliable (up to 98% faster uploads from our testing). The risk of corrupted artifacts is basically zero now as well. There is a computed sha256 output for each upload and we're even planning on adding it to our APIs in the future (for now you can see it outputted in the logs)

Here is just a random example of a 1GB artifact

https://gh.io/artifact-v3-vs-v4

Because of the major rearchitecture, uploads and downloads should no longer see any ECONNREFUSED or ECONNRESET errors!

Recommend switching to v4 today! 😃