search

ArchiveTeam / youtube-grab

Archiving all metadata from YouTube (everything except videos themselves due to size)
The Unlicense
23 stars 2 forks source link

429 on watch page doesn't trigger long wait #3

Closed JustAnotherArchivist closed 1 year ago

JustAnotherArchivist commented 3 years ago

Rate limits aren't only redirects to /sorry but can also be straight 429 responses. Those should trigger the long wait. (Hitting it rapidly will extend the ban otherwise.)

Example from knecht420 on IRC 17:59:03 < knecht420> task with 429 takes about 20 secs in total and then requeststs new one ``` Starting CheckIP for Item Checking IP address. Finished CheckIP for Item Starting GetItemFromTracker for Item Received item 'v1:lRnIPtB2SyY' from tracker Starting PrepareDirectories for Item v1:lRnIPtB2SyY Finished PrepareDirectories for Item v1:lRnIPtB2SyY Starting WgetDownload for Item v1:lRnIPtB2SyY Archiving item v1:lRnIPtB2SyY. 1=429 https://www.youtube.com/watch?v=lRnIPtB2SyY Server returned 429 (RETRFINISHED). Sleeping. 2=429 https://www.youtube.com/watch?v=lRnIPtB2SyY Server returned 429 (RETRFINISHED). Sleeping. 3=429 https://www.youtube.com/watch?v=lRnIPtB2SyY Server returned 429 (RETRFINISHED). Sleeping. I give up... Aborting item v1:lRnIPtB2SyY. Lua runtime error: youtube.lua:130: Aborting.. Process WgetDownload returned exit code -6 for Item v1:lRnIPtB2SyY Failed WgetDownload for Item v1:lRnIPtB2SyY Waiting 10 seconds... ```
Arkiver2 commented 3 years ago

thanks!

Arkiver2 commented 1 year ago

Fixed as of commit aaf61d346dbecf916906efeb4e4e0cd365e98efe (sleep of 1800 seconds).