search

simonw / package-stats

Download statistics for my PyPI packages
7 stars 1 forks source link

Rate-limiting error fetching PyPI stats #3

Closed simonw closed 3 years ago

simonw commented 3 years ago

httpx.HTTPStatusError: 429 Client Error: TOO MANY REQUESTS for url: https://pypistats.org/api/packages/healthkit-to-sqlite/overall?mirrors=true

simonw commented 3 years ago

https://github.com/crflynn/pypistats.org/issues/28#issuecomment-598417650

The rate limit is 5 per second, and 30 per minute.

https://github.com/crflynn/pypistats.org/blob/da4d4aa8da682047330dc8f6c5a8ea49c88e7905/pypistats/run.py#L23 confirms that is still in place.

simonw commented 3 years ago

I'm running against 102 packgase at the moment https://datasette.io/content?sql=select+substr%28nameWithOwner%2C+instr%28nameWithOwner%2C+%27%2F%27%29+%2B+1%29+from+datasette_repos%3B - so the 30 per minute thing is probably what I'm tripping here.

simonw commented 3 years ago

... but I have --sleep 10 which should keep me within that limit: https://github.com/simonw/package-stats/blob/f62bcc7720bdd9e6b125ee6734b14991afa86890/.github/workflows/fetch_stats.yml#L30