search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.91k stars 976 forks source link

E621 downloads are stored incorrectly (I think) in the SQL #2602

Closed DuendeInexistente closed 2 years ago

DuendeInexistente commented 2 years ago

I noticed g-dl was downloading things I'd already got, so I checked the archive file and... image It saves the search query as part of the file string, meaning if you find the same file in diferent queries it'll download the same file multiple times. Other boorus don't have this behavior, so it's either an oversight or devs with different ideas of how it should work.

mikf commented 2 years ago

You can use the archive-format option to overwrite the default und set a custom archive ID format string - just "{id}" is probably what you want.

Other boorus don't have this behavior

Most/All booru sites supported by gallery-dl have this behavior by default, because it would be strange and probably unwanted to skip most image downloads after downloading from a more general tag like "1girl" or "long_hair".

rautamiekka commented 2 years ago

Hmm, what if the keyword, in this case krisha_russel, is renamed and the old one doesn't redirect ?

DuendeInexistente commented 2 years ago

Oh, I see where the inconsistency lies now. I compared it to philomena boorus (IE Derpibooru) which I downloaded using /tag/ links. This wasn't a search query so that likely made archive format be indeed just {id}.

mikf commented 2 years ago

Hmm, what if the keyword, in this case krisha_russel, is renamed and the old one doesn't redirect ?

Well, that would invalid all previous archive entries for such a tag.

I agree in that default archive format should preferably not have included a search tag, but that default value was set many years ago (https://github.com/mikf/gallery-dl/commit/5008e105ee6d864ba91bd7b7532c72b97e9e7b1c) and I'd rather not break backwards compatibility here. Probably in 2.0, but not now.