search

backstrokeapp / server

:swimmer: A Github bot to keep repository forks up to date with their upstream.
https://backstroke.co
MIT License
667 stars 62 forks source link

Find a way to ensure that the queue doesn't reach critical mass #70

Closed 1egoman closed 7 years ago

1egoman commented 7 years ago

I've run into a few issues where the queue of link operations has reached a level where it is hard for the workers to recover, given the api rate limiting that github uses.

It'd be great if there was some way for the job that adds items to the queue to be sensitive of this and somehow not keep adding to the queue when it's really full.

1egoman commented 7 years ago

Within a day of deployment, the queue seems o be much more manageable. I suspect this was just sue to all links syncing for the first time initially, and now I probably see less than 100 link operations per hour. At that scale, I'm not worried about extra link operations finding their way into the queue.

Going to close for now, but will reopen if this becomes an issue again.