mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.97k stars 975 forks source link

Image Limits on exhentai & e-hentai #1288

Closed illudonite closed 1 year ago

illudonite commented 3 years ago

Hi All,

I'm still quite new to this. Basically just made an account to make this post. I'm not sure if this has already been reported / posted, but from a couple hours of testing today I found out a few things.

First, the facts: Every user on e/exhentai has a certain Image Limits on their account (max 5k for non-donors iirc) When loading a resampled image on e/exhentai, 1 image credit is consumed (after the first image, which consumes 5 for some reason).

What I found was that, when downloading with -o original=false, 2 image credits were consumed However, if I add -o skip=false to that line, only 1 image credits would be consumed

I'm have not looked into why this might be the case. I assume gallery-dl load some information / metadata once to check whether the file has already existed before proceeding to download the image. Sadly e/exhentai seems to have counted this as an attempt to download it instead, and so twice the image credits were consumed.

While the problem might seem minimal, if we combined this with the default config of original=true then the image credit will be consumed VERY quickly. (current normal rate is about 11 credits per MB of original image downloaded - a 36.7MB gallery I just downloaded consumed about 404 image limits with skip=false!).

If this is something that cannot be easily fixed (perhaps by means of keeping a separate logs for all the ex / ehentai gallery url requested (not just supplied) - i.e. doing the checks locally), then I would highly recommended adding skip=false as a default in e & exhentai config or else most people would probably hit their limit without even being able to download 10 galleries.

wankio commented 3 years ago

just use "archive": "~/gallery-dl/exhentai.sqlite3",

and --abort N Abort extractor run after N consecutive file downloads have been skipped, e.g. if files with the same filename already exist