search

crflynn / pypistats.org

PyPI downloads analytics dashboard
https://pypistats.org/
140 stars 10 forks source link

502, 504 or slow response on all requests #20

Closed orsinium closed 4 years ago

orsinium commented 5 years ago

Don't die, please :heart:

crflynn commented 5 years ago

Looks like the API is being spammed right now by someone using python-requests. I didn't design this application's infrastructure to scale to any degree but I'll look into what I can do to remedy this.

hugovk commented 5 years ago

https://github.com/hugovk/pypistats uses requests, but then requests is the 4th most downloaded package.

Would it help to add a User-Agent?

orsinium commented 5 years ago

Yeah, it could be anyone. However, it's a good reminder to set User-Agent. I've made an issue in my tool as well: https://github.com/dephell/dephell/issues/231

hugovk commented 5 years ago

PR to add User-Agent: https://github.com/hugovk/pypistats/pull/65.

crflynn commented 5 years ago

I've disabled the API for the time being. I don't have much time right now to try to resolve this but I don't want the service to be completely unusable for the community. The charts should be viewable for now at least.

calebcartwright commented 5 years ago

Hi there! Obviously we're big consumers of pypistats.org at Shields.io (we use Shields.io for user-agent) although based on the thread it doesn't sound like we're the ones causing the spam. Please let us know if we are though.

crflynn commented 5 years ago

While shields makes up a majority of the requests it was not what caused this issue. It was an individual who was trying to scrape the entirety of the API.

I'll reiterate to whomever that individual is that it is basically free to get all of this information directly from BigQuery (and in only a few seconds of query time). I'll reactivatee the API in the near future. Sorry for the disruption of service.

hugovk commented 5 years ago

PR to add User-Agent: hugovk/pypistats#65.

Merged and released in 0.7.0.