julia-actions / setup-julia

This action sets up a Julia environment for use in actions by downloading a specified version of Julia and adding it to PATH.
MIT License
93 stars 23 forks source link

[BUG] Intermittent Failure HTTP 403 #91

Closed Octogonapus closed 2 years ago

Octogonapus commented 2 years ago

Describe the bug We occasionally see an HTTP 403 error while running this action. The action fails after running for about 25 minutes. Rerunning the action usually resolves it, as this is an intermittent error.

To Reproduce Workflow file to reproduce the behavior:

      - uses: julia-actions/setup-julia@v1
        id: setup-julia
        with:
          version: 1.6

Expected behavior Since this is intermittent, I'd hope it's possible to have a retry mechanism in the setup action (maybe there is one but it just gets exhausted?) so that this error does not occur. It would also be useful to have some timeout on this action, since it typically takes a few seconds to run. It should certainly never run for more than a minute, let alone 25 minutes.

Screenshots/Build logs These are the logs I have on hand:

2021-12-15T19:47:36.1156784Z ##[group]Run julia-actions/setup-julia@v1
2021-12-15T19:47:36.1157340Z with:
2021-12-15T19:47:36.1157727Z   version: 1.6
2021-12-15T19:47:36.1158134Z   arch: x64
2021-12-15T19:47:36.1158607Z   show-versioninfo: false
2021-12-15T19:47:36.1159063Z env:
2021-12-15T19:47:36.1159478Z   JULIA_PKG_SERVER: 
2021-12-15T19:47:36.1159990Z   JULIA_CUDA_USE_BINARYBUILDER: true
2021-12-15T19:47:36.1160535Z   AWS_REGION: us-east-2
2021-12-15T19:47:36.1161040Z   AWS_DEFAULT_REGION: us-east-2
2021-12-15T19:47:36.1161690Z   SSH_AUTH_SOCK: /tmp/ssh-mQErVIAYT5Fr/agent.18054
2021-12-15T19:47:36.1162252Z   SSH_AGENT_PID: 18055
2021-12-15T19:47:36.1162692Z ##[endgroup]
2021-12-15T20:13:23.5119016Z ##[error]Unexpected HTTP response: 403

Additional context It's worth knowing that this is running on a self-hosted runner with a poor quality network connection. This error happens far more frequently on our self-hosted runners (with a bad network) than on hosted runners.

lamdor commented 2 years ago

I'm also seeing this error and just reported it on the julia slack: https://julialang.slack.com/archives/C68SPSE72/p1639603832138100. https://julialang-s3.julialang.org/bin/versions.json is returning a 403 and i think sometimes a runner has that cached is ok, otherwise, it fails and retries with the 403.

Octogonapus commented 2 years ago

This just started affecting all of our self-hosted runners. Some debug logs:

##[debug]Downloading https://julialang-s3.julialang.org/bin/versions.json
##[debug]Downloading /home/username/actions-runner/_work/_temp/fef20fec-795d-4feb-bb08-41e5394a8188
##[debug]Failed to download from "https://julialang-s3.julialang.org/bin/versions.json". Code(403) Message(Forbidden)
##[debug]Download of versions.json failed, trying again. Error: Error: Unexpected HTTP response: 403
##[debug]Downloading https://julialang-s3.julialang.org/bin/versions.json
##[debug]Downloading /home/username/actions-runner/_work/_temp/0978359c-21ab-470c-8ce1-59056a4b7e77
##[debug]Failed to download from "https://julialang-s3.julialang.org/bin/versions.json". Code(403) Message(Forbidden)
##[debug]Download of versions.json failed, trying again. Error: Error: Unexpected HTTP response: 403
##[debug]Downloading https://julialang-s3.julialang.org/bin/versions.json
##[debug]Downloading /home/username/actions-runner/_work/_temp/bd8ded02-890a-4ad6-b601-c7f13c763bbc
##[debug]Failed to download from "https://julialang-s3.julialang.org/bin/versions.json". Code(403) Message(Forbidden)
##[debug]Download of versions.json failed, trying again. Error: Error: Unexpected HTTP response: 403
##[debug]Downloading https://julialang-s3.julialang.org/bin/versions.json
##[debug]Downloading /home/username/actions-runner/_work/_temp/e3181214-e61d-4d3e-a071-9347abdcd80e
##[debug]Failed to download from "https://julialang-s3.julialang.org/bin/versions.json". Code(403) Message(Forbidden)
##[debug]Download of versions.json failed, trying again. Error: Error: Unexpected HTTP response: 403

Edit: This is affecting all of our runners, including GitHub hosted runners. Probably an AWS outage.

torfjelde commented 2 years ago

Seems like it's not related to the AWS issue: https://julialang.slack.com/archives/C68SPSE72/p1639609120139700?thread_ts=1639603832.138100&cid=C68SPSE72.

torfjelde commented 2 years ago

Seems to be fixed now.

SaschaMann commented 2 years ago

It's worth knowing that this is running on a self-hosted runner with a poor quality network connection. This error happens far more frequently on our self-hosted runners (with a bad network) than on hosted runners.

If the network connection is bad enough that you run into this outside of today's issue with AWS, let me know and I can adjust the retry settings.