harvard-lil / perma

Indelible links
420 stars 71 forks source link

Upload files to existing "daily" digest IA items, Part VI #3247

Closed rebeccacremona closed 1 year ago

rebeccacremona commented 1 year ago

Subsequent to Part V, this is now running steadily. We are monitored to see if too many confirmation tasks fail; if they do, we should consider running less frequently than every 15 min, and could also consider reducing the allowed number of retries (if we aren't seeing the retries succeed, then what's the point of trying more than once or twice?).

As a result, we can see that this util for checking ratelimiting is too slow, at this scale, because it is checking too many buckets.

We're going to try capping at 100, and see if that reduction is sufficient.

rebeccacremona commented 1 year ago

also consider reducing the allowed number of retries We decided to do this to: we aren't seeing any benefit from large numbers of retries.