search

foundry-rs / foundry

Foundry is a blazing fast, portable and modular toolkit for Ethereum application development written in Rust.
https://getfoundry.sh
Apache License 2.0
7.86k stars 1.57k forks source link

Foundryup failure due to GitHub API rate limiting #3942

Open frangio opened 1 year ago

frangio commented 1 year ago

Getting intermittent failures from foundryup in slither-action as seen here. The error doesn't seem abort the script and foundryup reports that it finished installing.

[-] Foundry target detected, installing foundry nightly
curl: (22) The requested URL returned error: 403 
foundryup: installing foundry (version nightly, tag nightly-)
foundryup: downloading latest forge and cast

######################################################################## 100.0%

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
foundryup: downloading manpages

######################################################################## 100.0%

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
foundryup: line 128: /opt/foundry/bin/forge: No such file or directory
foundryup: installed - 
foundryup: line 129: /opt/foundry/bin/cast: No such file or directory
foundryup: installed - 
foundryup: done

In the first two lines we see curl returns 403 status and the next line shows nightly-.

curl: (22) The requested URL returned error: 403 
foundryup: installing foundry (version nightly, tag nightly-)

What seems to be failing is this request:

https://github.com/foundry-rs/foundry/blob/f10df79e8d3a5e02eeac9bd0c4a4297d3d96bbae/foundryup/foundryup#L79-L84

Unauthenticated requests to the GitHub API are heavily rate limited:

For unauthenticated requests, the rate limit allows for up to 60 requests per hour. Unauthenticated requests are associated with the originating IP address, and not the person making requests.

This seems to make foundryup not well suited for CI environments in its current form. Perhaps it should use the nightly tag without a commit hash?

Consider using set -o pipefail in addition to set -e. I think this is why the error doesn't abort the script.

mattsse commented 1 year ago

my bashscript skills are pretty limited, but we def should handle this.

https://gist.github.com/mohanpedala/1e2ff5661761d3abd0385e8223e16425#set--o-pipefail

from what I understand adding set -o pipefail would be useful.

frangio commented 1 year ago

Yeah that should make the script at least catch this failure.

frangio commented 1 year ago

Hm, it seems like ensure should be doing that but it doesn't seem to be working as intended.

https://github.com/foundry-rs/foundry/blob/b78509fb8e11b7f58a0021cd900b7c7a2d3be503/foundryup/foundryup#L239-L244

sambacha commented 1 year ago

i tried to fix this but was closed https://github.com/foundry-rs/foundry/pull/2221

I actually have the fix here: https://github.com/foundry-rs/foundry/pull/3907/files#diff-79a844de4d6869f243ecd1e7ca4d9532d9a4987f85ca63d9f4af85aa227ecb38R22-R32

sambacha commented 1 year ago

Getting intermittent failures from foundryup in slither-action as seen here. The error doesn't seem abort the script and foundryup reports that it finished installing.

[-] Foundry target detected, installing foundry nightly
curl: (22) The requested URL returned error: 403 
foundryup: installing foundry (version nightly, tag nightly-)
foundryup: downloading latest forge and cast

######################################################################## 100.0%

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
foundryup: downloading manpages

######################################################################## 100.0%

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
foundryup: line 128: /opt/foundry/bin/forge: No such file or directory
foundryup: installed - 
foundryup: line 129: /opt/foundry/bin/cast: No such file or directory
foundryup: installed - 
foundryup: done

In the first two lines we see curl returns 403 status and the next line shows nightly-.

curl: (22) The requested URL returned error: 403 
foundryup: installing foundry (version nightly, tag nightly-)

What seems to be failing is this request:

https://github.com/foundry-rs/foundry/blob/f10df79e8d3a5e02eeac9bd0c4a4297d3d96bbae/foundryup/foundryup#L79-L84

Unauthenticated requests to the GitHub API are heavily rate limited:

For unauthenticated requests, the rate limit allows for up to 60 requests per hour. Unauthenticated requests are associated with the originating IP address, and not the person making requests.

This seems to make foundryup not well suited for CI environments in its current form. Perhaps it should use the nightly tag without a commit hash?

Consider using set -o pipefail in addition to set -e. I think this is why the error doesn't abort the script.

The real issue is that GitHub has no direct way to directly link to the latest build from GitHub actions of a given repository.

Even if you do have a link to an artifact, using it requires the visitor to be logged into the GitHub website.[^1]

[^1]: see https://github.com/actions/upload-artifact/issues/51

Because GitHub doesn't provide any permanent and public links to an artifact (this being the 'latest'), you should use a > service redirect to time-limited links that GitHub can give to the application -- only on behalf of an authenticated user that has access to the repository. So, whenever someone downloads an artifact from a repository that you had added, it would use a service token that is associated with your installation of the GitHub App. -- https://nightly.link/

Also, these are pre-releases so a service like https://nightly.link/ wont actually work

Here is a script we use to download the latest releases in json file, https://github.com/sambacha/foundry-release/blob/master/get-release.sh

just parse that to get your download link

collide143 commented 3 months ago

forge init hello_foundry && cd hello_foundry