[bug] Massive slowdown even when uploading artifacts (small total size)

RadxaYuntian commented 11 months ago

What happened?

Our workflow build some files into usable binaries, and upload them as artifacts. This usually completes in less than 2 minutes, but now they take more than an hours, and the additional time are all spent in this action.

There are also many 503 errors:

Run actions/upload-artifact@v3
  with:
    name: linux-rk356x
    path: arch/**/*.dtbo
    if-no-files-found: warn
With the provided path, there will be 277 files uploaded
Starting artifact upload
For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging
Artifact name is valid!
Container for artifact "linux-rk356x" successfully created. Starting upload of file(s)
Total file count: 277 ---- Processed file #14 (5.0%)
Total file count: 277 ---- Processed file #14 (5.0%)
Total file count: 277 ---- Processed file #14 (5.0%)
Total file count: 277 ---- Processed file #14 (5.0%)
Total file count: 277 ---- Processed file #14 (5.0%)
Total file count: 277 ---- Processed file #14 (5.0%)
Total file count: 277 ---- Processed file #14 (5.0%)
Total file count: 277 ---- Processed file #14 (5.0%)
Total file count: 277 ---- Processed file #14 (5.0%)
Total file count: 277 ---- Processed file #14 (5.0%)
Total file count: 277 ---- Processed file #14 (5.0%)
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #1. Waiting for 5940 milliseconds before continuing the upload at offset 0
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #1. Waiting for 5477 milliseconds before continuing the upload at offset 0
Finished backoff for retry #1, continuing with upload
Total file count: 277 ---- Processed file #21 (7.5%)
Finished backoff for retry #1, continuing with upload
Total file count: 277 ---- Processed file #24 (8.6%)
Total file count: 277 ---- Processed file #24 (8.6%)
Total file count: 277 ---- Processed file #24 (8.6%)
Total file count: 277 ---- Processed file #24 (8.6%)
Total file count: 277 ---- Processed file #24 (8.6%)
Total file count: 277 ---- Processed file #24 (8.6%)
Total file count: 277 ---- Processed file #24 (8.6%)
Total file count: 277 ---- Processed file #24 (8.6%)
Total file count: 277 ---- Processed file #24 (8.6%)
Total file count: 277 ---- Processed file #24 (8.6%)
Total file count: 277 ---- Processed file #24 (8.6%)
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #1. Waiting for 5837 milliseconds before continuing the upload at offset 0
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #1. Waiting for 4846 milliseconds before continuing the upload at offset 0
Finished backoff for retry #1, continuing with upload
Finished backoff for retry #1, continuing with upload
Total file count: 277 ---- Processed file #32 (11.5%)
Total file count: 277 ---- Processed file #41 (14.8%)
Total file count: 277 ---- Processed file #41 (14.8%)
Total file count: 277 ---- Processed file #41 (14.8%)
Exponential backoff for retry #2. Waiting for 10283 milliseconds before continuing the upload at offset 0
Total file count: 277 ---- Processed file #248 (89.5%)
Finished backoff for retry #1, continuing with upload
Finished backoff for retry #2, continuing with upload
Total file count: 277 ---- Processed file #251 (90.6%)
Total file count: 277 ---- Processed file #251 (90.6%)
Total file count: 277 ---- Processed file #251 (90.6%)
Total file count: 277 ---- Processed file #251 (90.6%)
Total file count: 277 ---- Processed file #251 (90.6%)
Total file count: 277 ---- Processed file #251 (90.6%)
Total file count: 277 ---- Processed file #251 (90.6%)
Total file count: 277 ---- Processed file #251 (90.6%)
Total file count: 277 ---- Processed file #251 (90.6%)
Total file count: 277 ---- Processed file #251 (90.6%)
Total file count: 277 ---- Processed file #251 (90.6%)
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #3. Waiting for 16620 milliseconds before continuing the upload at offset 0
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #1. Waiting for 5731 milliseconds before continuing the upload at offset 0
Total file count: 277 ---- Processed file #251 (90.6%)
Finished backoff for retry #1, continuing with upload
Total file count: 277 ---- Processed file #251 (90.6%)
Finished backoff for retry #3, continuing with upload
Total file count: 277 ---- Processed file #258 (93.1%)
Total file count: 277 ---- Processed file #258 (93.1%)
Total file count: 277 ---- Processed file #258 (93.1%)
Total file count: 277 ---- Processed file #258 (93.1%)
Total file count: 277 ---- Processed file #258 (93.1%)
Total file count: 277 ---- Processed file #258 (93.1%)
Total file count: 277 ---- Processed file #258 (93.1%)
Total file count: 277 ---- Processed file #258 (93.1%)
Total file count: 277 ---- Processed file #258 (93.1%)
Total file count: 277 ---- Processed file #258 (93.1%)
Total file count: 277 ---- Processed file #258 (93.1%)
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #2. Waiting for 9100 milliseconds before continuing the upload at offset 0
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #1. Waiting for 6199 milliseconds before continuing the upload at offset 0
Total file count: 277 ---- Processed file #258 (93.1%)
Finished backoff for retry #1, continuing with upload
Finished backoff for retry #2, continuing with upload
Total file count: 277 ---- Processed file #268 (96.7%)
Total file count: 277 ---- Processed file #268 (96.7%)
Total file count: 277 ---- Processed file #268 (96.7%)
Total file count: 277 ---- Processed file #268 (96.7%)
Total file count: 277 ---- Processed file #268 (96.7%)
Total file count: 277 ---- Processed file #268 (96.7%)
Total file count: 277 ---- Processed file #268 (96.7%)
Total file count: 277 ---- Processed file #268 (96.7%)
Total file count: 277 ---- Processed file #268 (96.7%)
Total file count: 277 ---- Processed file #268 (96.7%)
Total file count: 277 ---- Processed file #268 (96.7%)
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #1. Waiting for 5972 milliseconds before continuing the upload at offset 0
A 503 status code has been received, will attempt to retry the upload
Exponential backoff for retry #1. Waiting for 5237 milliseconds before continuing the upload at offset 0
Finished backoff for retry #1, continuing with upload
Finished backoff for retry #1, continuing with upload
Total size of all the files uploaded is 180607 bytes
File upload process has finished. Finalizing the artifact upload
Artifact has been finalized. All files have been successfully uploaded!

The raw size of all the files that were specified for upload is 494571 bytes
The size of all the files that were uploaded is 180607 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage

Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 

Artifact linux-rk356x has been successfully uploaded!

What did you expect to happen?

Take less than an hour to upload 500KB artifact.

How can we reproduce it?

https://github.com/radxa/overlays/actions

Workflows from 3 days ago are normal.

Anything else we need to know?

No response

What version of the action are you using?

v3

What are your runner environments?

linux

Are you on GitHub Enterprise Server? If so, what version?

No response

screig commented 11 months ago

https://github.com/actions/upload-artifact/issues/199

This issue ticket has been open since 2021, which details the same issue. The speed of this action and therefore the whole GitHub CI platform is a joke.

RadxaYuntian commented 11 months ago

I'm not sure if it's the same issue. This only happen recently to us.

gfellerph commented 11 months ago

Same here https://github.com/swisspost/design-system/actions/runs/5922729420/job/16059969235?pr=1833

djthornton1212 commented 11 months ago

Looks many of us are experiencing this according to #428 & #429.

PeelyZenobe commented 11 months ago

Throwing in my vote, uploading artifacts was working fine last week, but as of this morning, it's not. An upload of 2.5MB was taking 10 seconds on friday, and now it's take over 50 minutes.