Bionus / imgbrd-grabber

Very customizable imageboard/booru downloader with powerful filenaming features.
https://www.bionus.org/imgbrd-grabber/
Apache License 2.0
2.48k stars 216 forks source link

Cycle between connections during downloads #1500

Open Tokyo-All-Ready opened 5 years ago

Tokyo-All-Ready commented 5 years ago

as we are all aware sankakucomplex puts limits on the number of pictures you can download and if you grab too many (5+) it will limit you for a certain amount of time

can you add in a way to detect when the restriction is put on the ip and: download from another image batch while you wait out the time period or a mode to cycle threw proxy connections while the burned connection has to wait out its time restriction

attempt to download eww.jpg found restriction setting connection -> connection 2 download download download found restriction connection 2 -> connection 3 download download download fallback to connection 1

thats just my idea take it with a grain of salt im a scripter not a programmer

Bionus commented 5 years ago

can you add in a way to detect when the restriction is put on the ip and: download from another image batch while you wait out the time period

This could be done indeed, but not in the next version given it requires to re-think how batch downloads work. But instead of your suggestion to only switch to another batch when the first reaches the limit, I'd rather simply have one batch running per website, all of them in parallel.

or a mode to cycle threw proxy connections while the burned connection has to wait out its time restriction

No, this would be going against the wishes of the person who put the restriction in the first place, which is not what Grabber is for. 😄