search

Brawl345 / Get-DMAX-Links

Get links of Discovery shows and a specific season and/or episode
MIT License
21 stars 6 forks source link

Added 429 too many request detection #5

Closed vlcty closed 4 years ago

vlcty commented 4 years ago

The Discovery API has some rate limiting in place. I noticed it when attempting to fetch Moonshiners (ID 6009). After round about 40 episodes it started to throw HTTP 429 Too Many Requests responses. This resulted in skipped episodes in the xlsx file.

To counter this each episode is tried three times before skipping it. In case of a HTTP 429 response the application sleeps for a couple of seconds (1st attempt: 10 seconds, 2nd attempt 20 seconds, etc).

Sample output:

2020-09-08 23:29:47,747 - DMAX - INFO - Getting link 46 of 97 2020-09-08 23:29:47,747 - DMAX - INFO - Attempt 1 of 3 2020-09-08 23:29:47,901 - DMAX - INFO - Received 429 Too Many Requests. Waiting 10 seconds to cool down! 2020-09-08 23:29:57,904 - DMAX - INFO - Attempt 2 of 3 2020-09-08 23:29:58,070 - DMAX - INFO - Received 429 Too Many Requests. Waiting 20 seconds to cool down! 2020-09-08 23:30:18,075 - DMAX - INFO - Attempt 3 of 3 2020-09-08 23:30:18,256 - DMAX - INFO - Getting link 47 of 97 2020-09-08 23:30:18,256 - DMAX - INFO - Attempt 1 of 3

Thanks for this application btw!

Brawl345 commented 4 years ago

I will look into it when I have more time!