added tries and incremental delay

pawelsz-rb commented 1 year ago

Description of the change

Added 5 tries with incremental delay.

resty-go is using backoff method to increase retry intervals after each attempt. I think it's good enough for us, and it saves the development time.

Here are my 3 delay benchmarks using 8 seconds as the start, and 50 seconds as the max:

1:18PM DBG Users/pawel/go/pkg/mod/github.com/rs/zerolog@v1.20.0/log.go:390 > 8.001985834s
1:18PM DBG Users/pawel/go/pkg/mod/github.com/rs/zerolog@v1.20.0/log.go:390 > 13.578360833s
1:18PM DBG Users/pawel/go/pkg/mod/github.com/rs/zerolog@v1.20.0/log.go:390 > 22.769034625s
1:18PM DBG Users/pawel/go/pkg/mod/github.com/rs/zerolog@v1.20.0/log.go:390 > 38.384722458s

1:21PM DBG Users/pawel/go/pkg/mod/github.com/rs/zerolog@v1.20.0/log.go:390 > 8.002051416s
1:21PM DBG Users/pawel/go/pkg/mod/github.com/rs/zerolog@v1.20.0/log.go:390 > 11.219837459s
1:21PM DBG Users/pawel/go/pkg/mod/github.com/rs/zerolog@v1.20.0/log.go:390 > 21.410728333s
1:21PM DBG Users/pawel/go/pkg/mod/github.com/rs/zerolog@v1.20.0/log.go:390 > 39.316473042s

1:26PM DBG Users/pawel/go/pkg/mod/github.com/rs/zerolog@v1.20.0/log.go:390 > 8.001550333s
1:26PM DBG Users/pawel/go/pkg/mod/github.com/rs/zerolog@v1.20.0/log.go:390 > 15.233538917s
1:26PM DBG Users/pawel/go/pkg/mod/github.com/rs/zerolog@v1.20.0/log.go:390 > 19.549524041s
1:26PM DBG Users/pawel/go/pkg/mod/github.com/rs/zerolog@v1.20.0/log.go:390 > 47.789060292s

I think it's good enough and really close to what we agreed

in addition, I have set additional timeout for http client (30 seconds).

That way it will give us plenty of time to recover. Also I don't want to recover errors yet (returning success in case of failure). Let'e see first if this change will help.

Type of change

[ ] Bug fix (non-breaking change that fixes an issue)
[ ] New feature (non-breaking change that adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Maintenance
[ ] New release

Related issues

Shortcut stories and GitHub issues (delete irrelevant)

Fix #318

Checklists

Development

[x] Lint rules pass locally
[x] The code changed/added as part of this pull request has been covered with tests
[x] All tests related to the changed code pass in development

Code review

[x] This pull request has a descriptive title and information useful to a reviewer. There may be a screenshot or screencast attached
[x] "Ready for review" label attached to the PR and reviewers assigned
[x] Issue from task tracker has a link to this pull request
[x] Changes have been reviewed by at least one other engineer

pawelsz-rb commented 1 year ago