pulumi / pulumi-kubernetes-operator

A Kubernetes Operator that automates the deployment of Pulumi Stacks
Apache License 2.0
226 stars 54 forks source link

[v2] Retry failed updates with exponential backoff #709

Closed blampe closed 1 month ago

blampe commented 1 month ago

Currently, if our automation APIs call fail they return non-nil errors to the operator. In #676 I modified Update to translate these errors into a "failed" status on the Update/Stack, but other operations (preview etc.) still surface these errors and automatically re-queue.

We'd like to retry these failed updates much less aggressively than we retry transient network errors, for example. To accomplish this we do a few things:

Fixes #677

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 78.17259% with 43 lines in your changes missing coverage. Please review.

Project coverage is 53.68%. Comparing base (83f8438) to head (f15ea11). Report is 1 commits behind head on v2.

Files with missing lines Patch % Lines
...ator/internal/controller/auto/update_controller.go 71.11% 36 Missing and 3 partials :warning:
...tor/internal/controller/pulumi/stack_controller.go 93.22% 3 Missing and 1 partial :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## v2 #709 +/- ## ========================================== + Coverage 50.22% 53.68% +3.46% ========================================== Files 27 27 Lines 2919 2902 -17 ========================================== + Hits 1466 1558 +92 + Misses 1272 1164 -108 + Partials 181 180 -1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.