Upload consistently times out after hitting HTTP 503 several times

[rectified95]

What happened?

We’re consistently hitting timeouts and HTTP 503 errors during artifact upload from our self-hosted runners. The artifact consists of a 600MB binary file and a small text file. The task times out after about 35mins, and only the text file gets uploaded. (Compressing the 600MB file to 70MB doesn’t solve it.) At the same time, a manual download of the large file from the runner host to my dev box takes 3 minutes and succeeds. Snippet of the logs:

Finished backoff for retry #5, continuing with upload
Total file count: 3 ---- Processed file #2 (66.6%)
Total file count: 3 ---- Processed file #2 (66.6%)
Total file count: 3 ---- Processed file #2 (66.6%)
Total file count: 3 ---- Processed file #2 (66.6%)
A 503 status code has been received, will attempt to retry the upload
##### Begin Diagnostic HTTP information #####
Status Code: 503
Status Message: Service Unavailable
Header Information: {
  "content-length": "325",
  "content-type": "application/json; charset=utf-8",
  "date": "Fri, 13 Oct 2023 14:36:15 GMT",
  "server": "Kestrel",
  "cache-control": "no-store,no-cache",
  "pragma": "no-cache",
  "strict-transport-security": "max-age=[259](https://github.com/microsoft/ebpf-for-windows/actions/runs/6508676858/job/17679087053?pr=2978#step:38:260)2000",
  "x-tfs-processid": "5ae7dd07-8bd6-4f88-8170-4b3c1369b009",
  "activityid": "5456abeb-7a74-4302-abf5-fd66ab5cb412",
  "x-tfs-session": "5456abeb-7a74-4302-abf5-fd66ab5cb412",
  "x-vss-e2eid": "5456abeb-7a74-4302-abf5-fd66ab5cb412",
  "x-vss-senderdeploymentid": "ac98198d-4d34-0364-420d-bafa6e51dce2",
  "x-frame-options": "SAMEORIGIN"
}
###### End Diagnostic HTTP information ######
Retry limit has been reached for chunk at offset 0 to https://pipelinesghubeus3.actions.githubusercontent.com/eYokHsiyUgkTvpuaVsIXahob1lqD1Q84LS3b9LsFqZB2ZffrLc/_apis/resources/Containers/21519443?itemPath=Test-Logs-km_mt_stress_tests_restart_extension-x64-Debug%5Cvm4_ws2019%5CKernelDumps%5Ckm_dumps.zip
Warning: Aborting upload for C:\actions_runner_2019_4\_work\ebpf-for-windows\ebpf-for-windows\x64\Debug\TestLogs\vm4_ws2019\KernelDumps\km_dumps.zip due to failure

What did you expect to happen?

Artifacts well below the size limit get uploaded.

How can we reproduce it?

eBPF for Windows nightly CI pipeline, step "Upload Log Files" https://github.com/microsoft/ebpf-for-windows/actions/runs/6466462711/job/17555543203

Anything else we need to know?

No response

What version of the action are you using?

latest

What are your runner environments?

self-hosted

Are you on GitHub Enterprise Server? If so, what version?

No response

[Kruemmelspalter]

I have the same problem, it usually happens every three to four builds which is very annoying

[konradpabjan]

v4 upload-artifact was released today: https://github.blog/changelog/2023-12-14-github-actions-artifacts-v4-is-now-generally-available/

Recommend switching to it as it is powered by an entirely new backend and these 503s should no longer happen at all