Closed Groxx closed 2 years ago
Files with Coverage Reduction | New Missed Lines | % | ||
---|---|---|---|---|
internal/compatibility/thrift/history.go | 11 | 57.65% | ||
<!-- | Total: | 11 | --> |
Totals | |
---|---|
Change from base Build 01816e39-350f-43ae-a79e-40eaad99261d: | 0.1% |
Covered Lines: | 12404 |
Relevant Lines: | 19429 |
On second thought: nah, allow retries on panics. If they have retries configured, they'll have retries, and the onus is on them to include errRetryPanic
in the non-retryable list.
Previously, a local activity that panicked would lead to a panic exiting the goroutine that performed the call, terminating the whole worker process immediately.
Obviously this is not good behavior. So now that's fixed.
Seems like it has been this way since local activities were introduced, in commit 79c072aea60cbf2ed9bfc47ca85e793a8c07d77b , so it's pretty surprising that it went unnoticed / unfixed for this long. Particularly since it stands a decent chance of killing every worker instance due to decision task retries.
I'm not quite confident that this is done, but I'm putting it up for early review / eyeballs / tests while I read more of the surrounding code.
In particular, a simple and non-breaking (because this was always broken) change that may be worth making is to not do retries on local activities that panic:
Since panics are inherently a bit more dangerous / imply a more significant violation of expectations, perhaps it's worth treating them as fatal?