Strategy for dealing with the reddit API rate limit

[vitosamson]

As more subs start using modlogs, it's getting harder to stay under the reddit rate limit. This is especially prevalent during the initial import of a sub's logs, I think, because of the massive amount of requests needed to fetch thousands of logs.

Some quick thoughts:

• If a job fails, it must be removed from the queue. Otherwise it just sits there inactive but subsequent jobs for the same subreddit will not start because they think the job is already in progress.
• If a job fails, requeue it with a delay.
• Set a limit on the number of logs to fetch from reddit. If we fetch only e.g. 1000 at a time, it'll take longer to fetch all of them, but we may not hit the rate limit.

[justcool393]

One thing that may be useful that each request for the /about/log can take up to 500 (with ?limit=500) instead of the usual 100. This gets a whole lot more for each request.

[xaur]

?limit=500 is great for initial sync. Mind that Reddit also has some limit how far back you can go (not sure if it is based on time or the number of items).

For incremental syncs I would use a smaller limit but it may depend on how active the sub is.

Tracked subs with low activity can be checked less frequently. For example, you can compute a simple average activity based on past N days and estimate how often to poll it again and/or the limit param.