mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.81k stars 968 forks source link

[kemono.party] avoid downloading duplicate files on fantia #3423

Open RJFAC opened 1 year ago

RJFAC commented 1 year ago

Is it possible to avoid downloading duplicate files on fantia on kemono.party? eg.https://kemono.party/fantia/user/34875 It is kinda inconsistent for each fantia post, some posts doesn't dupes at all, some posts only dupes part of the files, others dupes the whole post.

mikf commented 1 year ago

You should be able to achieve this with a download archive that stores file hashes as key.

"kemonoparty": {
    "archive": "path/to/archive.sqlite",
    "archive-format": "{hash}"
}

edit: see also #2811

baraghiffari commented 1 year ago

You should be able to achieve this with a download archive that stores file hashes as key.

"kemonoparty": {
    "archive": "path/to/archive.sqlite",
    "archive-format": "{hash}"
}

edit: see also #2811

i tried changing it but the files i downloaded now goes to default user folder, not the ones where i set on basedirectory setting

Hrxn commented 1 year ago

What? Changing archive settings does not change your directory settings?

afterdelight commented 1 year ago

show full verbose log

baraghiffari commented 1 year ago

I tried using verbose. turns out it was missing "," on that "{hash}" coz there's another line below, it's now goes to correct folder. Thanks

RJFAC commented 1 year ago

You should be able to achieve this with a download archive that stores file hashes as key.

"kemonoparty": {
    "archive": "path/to/archive.sqlite",
    "archive-format": "{hash}"
}

edit: see also #2811

It won't work, because the duplicates are not in the same resolution.