search

Bionus / imgbrd-grabber

Very customizable imageboard/booru downloader with powerful filenaming features.
https://www.bionus.org/imgbrd-grabber/
Apache License 2.0
2.56k stars 219 forks source link

Gelbooru and rule34 download protection/limit #322

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 9 years ago

Gelbooru and rule34.paheal.net (not rule34.booru.org) have both some kind of download 'protection'. They are different on each site.

Gelbooru

After viewing (or downloading images via grabber) about 10 images, gelbooru send the user to a commercial ads page. There, the user must wait for 10 seconds before clicking on the link to see the image, and continue using gelbooru as normal for apparently 24 hours. The problem with grabber is that it does not 'see' the ads, and pass over this commercial 'wall'. That would be fine, except that, once the ads page as been activated, but not seen, and passed normally on a web browser, gelbooru stop sending information needed for tokens like %artist%, %copyright% and %character%. The result for those ads will always give a void answer (unknown, anonymous). The solution might be a message error with automated pause, inviting the user to go on the website and activate the ads.

rule34.paheal.net

It seem to be the simple usual 'vacuum software' protection that many images websites got (like e-hentai), to avoid abusive large amount of fast data download at once; something that grabber does.^^ after about 60~70 images downloaded, grabber simply hang. Your IP is locked for access to the website, by any means (also on web browser). I do not know exactly how much time must be waited beforebeing able to access rule34.paheal.net again, but it might be over 30 minutes. For those kind of boorus (even if rule34.paheal.net is the only one I know that use that protection) there should be a limitation feature. I suppose that if images were downloaded more slowly, and one by one, the protection will not activate, and the IP will not be kicked. Even if it slow the process, it will be better to have slow download, that have them interrupted after 40 seconds of download, and have to wait for 30 minutes before continuing, manually.

PS: btw, dont you have a e-hentai grabber^^???

Original issue reported on code.google.com by ser...@hotmail.com on 16 Jul 2014 at 5:58

GoogleCodeExporter commented 9 years ago

I'd like to add that rule34.paheal.net for me at least locks up when it scans the pages to get the image links. As long as I have it to download 1by1, it usually locks up. So I to vote to get at least a "page grab limiter"

Original comment by reitheki...@gmail.com on 10 Jan 2015 at 12:54

Bionus commented 8 years ago

For Gelbooru, see issue #270.

For rule34, a limiter has been added a few versions ago to limit the speed of downloading pages and images, you can access it from the sources settings.