ci: Increase max retry for the sweeper

[engedaam]

Fixes #N/A

Description

• We are experiencing throttling in our sweeper and will need to increase the default retry attempt

  operation error IAM: ListInstanceProfileTags, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: f4676348-05c7-47e8-aae5-6402e1ce5040,

• Since the retry strategy is exponential backoff, increasing the number of retry attempts from 3 to 10 to make sure that call will be spread out exponentially and increase the likelihood of call succeeding

How was this change tested?

Does this change impact docs?

• [ ] Yes, PR includes docs updates
• [ ] Yes, issue opened: #
• [x] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[netlify[bot]]

✅ Deploy Preview for karpenter-docs-prod canceled.

Name	Link
🔨 Latest commit	b29caec865e717bd5d20ca43a87692136a7aadde
🔍 Latest deploy log	https://app.netlify.com/sites/karpenter-docs-prod/deploys/6671995c1e61b10008c0fe01

[coveralls]

Pull Request Test Coverage Report for Build 9566930979

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 1 unchanged line in 1 file lost coverage.
• Overall coverage decreased (-0.01%) to 82.467%

---

Files with Coverage Reduction	New Missed Lines	%
pkg/providers/amifamily/ami.go	1	90.56%
<!--	Total:	1		-->

Totals
Change from base Build 9556663188:	-0.01%
Covered Lines:	5550
Relevant Lines:	6730

---

💛 - Coveralls

[rschalo]

What's the schedule for the sweeper again? Why 10 as the max?

[engedaam]

The schedule the sweeper is every 12 hours. Giving 10 max retries allows us to attempt to get instance profiles tags with more spread of a delay to allow us to refile the read tokens. The schedule will be as such:

0 | 0.030 | 1 | 0.090 | 2 | 0.210 | 3 | 0.450 | 4 | 0.930 | 5 | 0.150 | 6 | 0.180 | 7 | 0.210 | 8 | 0.240 | 9 | 0.270 |

Updated 1 0.03 2 0.09
3 0.21
4 0.45
5 0.93
6 1.89
7 3.81
8 7.65
9 15.33
10 30.69

[jmdeal]

Is the backoff linear or exponential? I'm reading the schedule you wrote as "incremented by 30ms each time" rather than an exponential backoff. I don't think it matters too much, but wanted to clarify before I approve. Otherwise looks good.

[engedaam]

I fixed the comment, the backoff is exponential