search

demophoon / trello-concert-tracker

Track concerts and post them to trello
0 stars 1 forks source link

Sync speed is fairly slow #20

Closed rattboi closed 7 years ago

rattboi commented 7 years ago

It seems like the last.fm API calls are pretty slow, and it looks intentional. I see that there is a 3-second pause between each call.

According to https://www.last.fm/api/tos Clause 4.4, we should be able to have 5 API calls per second, so a sleep of 0.2 should basically realize that. If we wanted to be more safe, we could do a 0.3 second pause.

Is there something I'm missing here?

rattboi commented 7 years ago

For the scraping part, how about using https://github.com/kennethreitz/grequests or https://github.com/ross/requests-futures to parallelize the operations?

rattboi commented 7 years ago

Looks like the limit for Trello is defined here: http://help.trello.com/article/838-api-rate-limits, which breaks down to 10 requests per second. Probably a 5 second sleep isn't necessary?

demophoon commented 7 years ago

Yeah, the pauses were intentional because I had been running into some rate limit issues and couldn't find anything on last.fm's site relating to their limits. Thanks for pulling that up, i'd be down for decreasing those delays.