excessive retries and backoff times?

[slinkp]

The exponential backoff time seems rather excessive with the default MAX_ATTEMPTS of 25 This line:

backoff = timedelta(seconds=(self.attempts \ 4) + 5)

results in multi-minute rescheduling past the third attempt, and hours past the eighth attempt, ending at several days. For example:

for i in range(25): print i, "attempts; minutes =", (i \ 4 + 5) / 60.0 ... 0 attempts; minutes = 0.0833333333333 1 attempts; minutes = 0.1 2 attempts; minutes = 0.35 3 attempts; minutes = 1.43333333333 4 attempts; minutes = 4.35 5 attempts; minutes = 10.5 6 attempts; minutes = 21.6833333333 7 attempts; minutes = 40.1 8 attempts; minutes = 68.35 # An hour 9 attempts; minutes = 109.433333333 10 attempts; minutes = 166.75 11 attempts; minutes = 244.1 12 attempts; minutes = 345.683333333 13 attempts; minutes = 476.1 14 attempts; minutes = 640.35 15 attempts; minutes = 843.833333333 16 attempts; minutes = 1092.35 17 attempts; minutes = 1392.1 # Almost a day 18 attempts; minutes = 1749.68333333
19 attempts; minutes = 2172.1 20 attempts; minutes = 2666.75 21 attempts; minutes = 3241.43333333 22 attempts; minutes = 3904.35 23 attempts; minutes = 4664.1 24 attempts; minutes = 5529.68333333 # Almost 4 days!

The reschedule deltas aren't logged either, and neither are exceptions, so it's easy to have a trivial mistake such as a NameError in your background function and have it rescheduled for days, and unless you read the django-background-task source and do the math, you won't realize it's doing that; and unless you do your own logging you won't ever learn why it fails.

[lilspikey]

Yeah, I probably need to add some documentation about all of this. The backoff logic is from delayed job (this is a rough port of it), but they do at least mention specific values in their docs.

Logging exceptions etc makes sense. I was relying on looking at the database last_error column for this, but no reason why it can't be logged too.

Will add more docs and logging later.

John

[lilspikey]

Ok have updated docs and added more logging now.