search

openSNP / snpr

The sources of the openSNP website
http://opensnp.org
MIT License
170 stars 46 forks source link

Add sidekiq-throttler to sidekiq jobs involving API #256

Open philippbayer opened 8 years ago

philippbayer commented 8 years ago

As in this comment thread: https://github.com/openSNP/snpr/pull/251#discussion_r52427780

Use https://github.com/gevans/sidekiq-throttler for all API jobs to be nicer :sparkles: :sparkling_heart: :sparkles:

PLOS has 7200 requests per 24 hours, Mendeley has 500/hour maybe outdated source, Fitbit 150/hour source, SNPedia ???, but something much less

Edit: Looks like sidekiq-throttler only supports Sidekiq 2/3, not 4!

gedankenstuecke commented 8 years ago

This would address #96 as well I guess? :closed_book:

philippbayer commented 8 years ago

Yes I'd hope so :D

However, sidekiq-throttler looks like it's not supported anymore. sidekiq-rate-limiter looks a bit wonky and unsupported, too. There's a sidekiq4 branch which looks like it works: https://github.com/Docady/sidekiq-rate-limiter/tree/sidekiq-4

tsujigiri commented 8 years ago

Sidekiq Enterprise has Rate Limiting. No clue how much this would cost... We can ask them, though.

philippbayer commented 8 years ago

One problem is that sidekiq-limit_fetch, which we're currently using, is incompatible with the other rate-limiting gems, since they all patch into the way Sidekiq retrieves jobs from the queue.

One workaround is to remove sidekiq-limit_fetch and use the time-limits for non-API jobs like parsing, too. Yay/Nay?