i got an error in my logs from this and it generated an alert, so i looked into it
my theory is that, my timeouts are on the order of milliseconds, but i have full jitter (the default), and by chance the resulting backoff computed was very small, like microseconds. then because retry policy returns a datetime instead of a duration, we end up computing Utc::now (a second time) to go back to a duration, and this second time the result was negative because some time had passed.
instead of returning a fatal error in this case, we should just continue retrying imo, and sleeping for 0 seconds seems fine.
i got an error in my logs from this and it generated an alert, so i looked into it
my theory is that, my timeouts are on the order of milliseconds, but i have full jitter (the default), and by chance the resulting backoff computed was very small, like microseconds. then because retry policy returns a datetime instead of a duration, we end up computing Utc::now (a second time) to go back to a duration, and this second time the result was negative because some time had passed.
instead of returning a fatal error in this case, we should just continue retrying imo, and sleeping for 0 seconds seems fine.
lmk what you think, i know this is very niche