Question - am I sending too many concurrent requests?

ecosyste-ms / packages

An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

https://packages.ecosyste.ms

GNU Affero General Public License v3.0

25 stars 4 forks source link

Question - am I sending too many concurrent requests? #650

Open jamietanna opened 7 months ago

jamietanna commented 7 months ago

I've noticed that every so often - particularly when I was testing with a much higher concurrent request count (~250 goroutines all sending off requests) - I would receive HTTP 500s from Ecosystems.

What's the recommendation for how many concurrent requests I should be trying to keep to?

Upvote & Fund

We're using Polar.sh so you can upvote and help fund this issue.
We receive the funding once the issue is completed & confirmed by you.
Thank you in advance for helping prioritize & fund our backlog.

andrew commented 7 months ago

Yeah you are likely DOSing the service! We don’t have a lot of funding for infrastructure so there’s limits to what we can provide right now.

There may be better ways to query the API based on the shape of the data you are trying to extract.

jamietanna commented 6 months ago

Sorry for the delay - thank you, yeah I'm gonna drop down the rate I hit Ecosystems, as well as make it so there's a little more caching in place locally to try and avoid a lot of unnecessary requests :crossed_fingers: