connor4312 / cockatiel

🐦 A resilience and fault-handling library. Supports Backoffs, Retries, Circuit Breakers, Timeouts, Bulkhead Isolation, and Fallbacks.
MIT License
1.59k stars 50 forks source link

decorrelatedJitterGenerator returns NaN #86

Closed novemberborn closed 6 months ago

novemberborn commented 10 months ago

Perhaps I'm using it wrong, but after 1025 attempts the jitter in the state goes to Infinity and for the 1026th attempt the delay is NaN.

const { decorrelatedJitterGenerator: d }= require('cockatiel')

const loop = (limit, state = [0,0]) => {
  for(let i = 0; i < limit; i++) {
    let delay;
    ([delay, state] = d(state, { generator: d, exponent: 2, maxDelay: 15*60_000, initialDelay: 45_000 }));
    console.log(delay, state)
  }
}

loop(1026)

Output:

…
900000 [ 1023, 5.201197568175768e+307 ]
900000 [ 1024, 1.3382550381451475e+308 ]
900000 [ 1025, Infinity ]
NaN [ 1026, Infinity ]
mpareja commented 6 months ago

We have encountered a CRITICAL failure where-in retries are performed without delay. The issue is likely related to the problem outlined in this ticket since the issue occurs after 1025 retries.

The problem can be reproduced using a retry with new ExponentialBackoff({ initialDelay: 1_000, maxDelay: 100 }). After ~1025 attempts, the retries occur much more aggressively than every 100ms.

The issue is critical because the retry loop has the potential to overwhelm downstream systems.

mpareja commented 6 months ago

@novemberborn Maybe we should update the title in this ticket to reflect the criticality? Something like, CRITICAL: error state results in retries without delay

connor4312 commented 6 months ago

Fixed in 3.1.3