Cache receiving stalled

actions / runner

The Runner for GitHub Actions :rocket:

https://github.com/features/actions

MIT License

4.77k stars 932 forks source link

Cache receiving stalled #2146

Open AlexeyPechnikov opened 1 year ago

AlexeyPechnikov commented 1 year ago

The same log line repeating indefinitely for 40+ minutes: Received 1653791515 of 1662180123 (99.5%)

    - name: Cache dataset
      uses: actions/cache@v3
      with:
        path: tests/S1A_Stack_CPGF_T173.tar.gz
        key: S1A_Stack_CPGF_T173.tar.gz
        restore-keys: S1A_Stack_CPGF_T173

This is the link to the job: https://github.com/mobigroup/gmtsar/actions/runs/3095306955/jobs/5009583727

AvaStancu commented 1 year ago

Hi @mobigroup ! Thank you for flagging this issue. The logs linked are not available anymore. Is this issue still happening?

julianiag commented 1 year ago

@AvaStancu , this issue is still happening from the private org repos from what I can see

julianiag commented 1 year ago

Could we remove the awaiting-customer-reponse label from the ticket as we can still confirm the issue is still not resolved.

thomasguerneyiag commented 1 year ago

This is a considerable issue in our self-hosted runners. Is there any update on this?

fhammerl commented 1 year ago

Hi @thomasguerneyiag,

do you have a way to reproduce this issue consistently or is it intermittent?
Is it only cache or other actions too? (e.g artifact-download)
Does it eventually resolve?
Would you mind dumping all the logs from the job where cache hangs like that? (Worker_20230412-HHmmSS-utc.log)

czeslavo commented 1 year ago

We've got a very similar issue in our workflows. We use setup-go@v4 (in this workflow) that sometimes stales on cache download percentage that's near 100% (example run - the job was running on a hosted ubuntu-latest runner).

According to my quick investigation, it seems that the issue may probably lie somewhere in downloadCacheStorageSDK function from actions/toolkit. I assume that because:

setup-go action uses @actions/cache's restoreCache
restoreCache calls cacheHttpClient.downloadCache that may call either downloadCacheStorageSDK or downloadCacheHttpClient, but in our case StorageSDK variant is most probably used because only it uses the DownloadProgress with the output we're seeing.

@fhammerl I'm not sure if that's reproducible, for us it happens quite randomly. It doesn't resolve eventually - it stales forever until the runner timeout is triggered and fails the job. I attach the logs from our failing job.

pct-cclausen commented 1 year ago

I am seeing this as well, it is a major issue in my workflow, the primary reason why it is very unreliable.

Patric-Sandberg-work commented 1 year ago

We are seeing this issue as well at random times.

fxnn commented 8 months ago

actions/cache introduced a timeout (defaulting to 10min) in version 3.0.8, see the documentation.

MartinNowak commented 6 months ago

Still saw this with a github managed runner ubuntu-20.04.6 and actions/cache@v4.