solo-io / go-utils

golang utilities
Apache License 2.0
111 stars 19 forks source link

Github upload release asset retry logic doesn't handle http 500s #394

Closed kdorosh closed 4 years ago

kdorosh commented 4 years ago

Sample builds:

The error:

{"level":"fatal","ts":"2020-02-21T19:21:27.235Z","caller":"githubutils/upload_release_asset.go:83","msg":"Error uploading assets. Error was: All attempts fail:
#1: POST https://uploads.github.com/repos/solo-io/gloo/releases/23887513/assets?name=glooctl-darwin-amd64: 500 Error saving asset []
#2: stat _output/glooctl-darwin-amd64: use of closed file
#3: stat _output/glooctl-darwin-amd64: use of closed file
#4: stat _output/glooctl-darwin-amd64: use of closed file
#5: stat _output/glooctl-darwin-amd64: use of closed file
#6: stat _output/glooctl-darwin-amd64: use of closed file
#7: stat _output/glooctl-darwin-amd64: use of closed file
#8: stat _output/glooctl-darwin-amd64: use of closed file
#9: stat _output/glooctl-darwin-amd64: use of closed file
#10: stat _output/glooctl-darwin-amd64: use of closed file","stacktrace":"github.com/solo-io/go-utils/githubutils.uploadFileOrExit
\t/go/pkg/mod/github.com/solo-io/go-utils@v0.13.1/githubutils/upload_release_asset.go:83
github.com/solo-io/go-utils/githubutils.uploadReleaseAssetOrExit
\t/go/pkg/mod/github.com/solo-io/go-utils@v0.13.1/githubutils/upload_release_asset.go:51
github.com/solo-io/go-utils/githubutils.uploadReleaseAssetsOrExit
\t/go/pkg/mod/github.com/solo-io/go-utils@v0.13.1/githubutils/upload_release_asset.go:42
github.com/solo-io/go-utils/githubutils.UploadReleaseAssetCli
\t/go/pkg/mod/github.com/solo-io/go-utils@v0.13.1/githubutils/upload_release_asset.go:37
main.main
\t/workspace/gloo/ci/upload_github_release_assets.go:70
runtime.main
\t/usr/local/go/src/runtime/proc.go:203"}

Note that this means a single HTTP 500 is enough for our entire retry loop to fail.

We should be able to upload release assets even if Github is experiencing intermittent failures.