search

mozilla / api.webmaker.org

Services for Webmaker
https://api.webmaker.org
17 stars 14 forks source link

Extend pagination #188

Closed xmatthewx closed 6 years ago

xmatthewx commented 8 years ago

I built a new tool for browsing potentially interesting projects. But, I can only search the 5,000 most recent projects, leaving 45,000 in the dark. Can we increase pagination or the max projects for a single query?

http://xmatthewx.github.io/webmaker-firehose/

cadecairos commented 8 years ago

we'll take a huge performance hit on queries beyond page 50 (offset > 5000). It may not be a problem initially but could be a DOS attack vector if requests pile up

We'd want to consider adding rate limiting in that case. I had a rate limit patch implemented months ago but it never got reviewed.

cadecairos commented 8 years ago

@xmatthewx I'm doing some issue housekeeping - is this still a thing we should do?

xmatthewx commented 8 years ago

I'm doing some issue housekeeping

I noticed. 😄

I actually need to put together a report on Webmaker. This could be helpful, but only if it's an easy change. If we throttle requests, higher pagination doesn't seem too risky. Again, don't do this if it'll take more than a few minutes.

In related news, it looks like the certificate for api.webmaker.org has expired. Should I file that, or is that already in progress?

cadecairos commented 6 years ago

This isn't going to happen.