search

watsonbox / exportify

Export/Backup Spotify playlists using the Web API
https://exportify.app/
MIT License
3.15k stars 436 forks source link

Manage the amount requests to the Web API to avoid rate limiting #12

Closed JMPerez closed 3 years ago

JMPerez commented 9 years ago

I think you could use something like https://github.com/JMPerez/promise-throttle to limit how many requests per second are made. You have an example of how this is done on https://github.com/JMPerez/spotify-dedup/blob/master/app/scripts/main.js

If you like the idea I can work on a Pull Request to add this.

watsonbox commented 9 years ago

Yes, I like this idea. I guess it would only ensure no throttling errors when using a personal application though, since according to the docs "Rate limiting is applied on an application basis (based on client id), regardless of how many users are using it."

I'm going to add a progress bar for the bulk export soon, too, which would be a pre-requisite for adding this kind of slowed down exporting I think.

JMPerez commented 9 years ago

You can still have rate limiting issues if several users are using the site at the same time, but you would have more control on the amount of requests a given user makes per unit of time.

pavelkomarov commented 5 years ago

My version implements rate limiting, 10 queries per second to the server. This is essential to make exporting really large playlists or all together work. Even for users with insanely large collections, it works in only a few minutes. Progress bar not super necessary.

I've hard-coded my client id, so everyone who uses it racks up traffic under the same limit. So if lots of users at once happen to all try to Export All at the same time, then yes they'll still run in to rate limiting problems. But this is unlikely until there are thousands of users every day, so I've chosen to just ignore it. If we ever get to that point, then I'll have to do something like give everyone their own client id.

exportify.net

watsonbox commented 4 years ago

Terribly overdue 🙈 but I'll see what I can do in updating this repo to support this and using as much of the work that's been done as possible. Thank you @pavel-aicradle for advancing it. I think this issue will best serve as the main one for dealing with this, put there's a lot of useful discussion in these:

pavelkomarov commented 4 years ago

70 gives you my changes, which include the fix to this.

watsonbox commented 3 years ago

Should now be fixed by https://github.com/watsonbox/exportify/pull/75