search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.79k stars 966 forks source link

[nijie] Skipping issues #6097

Open reyaz006 opened 2 months ago

reyaz006 commented 2 months ago 
"extractor": {
        "skip": true,
        "nijie": {
            "archive": "nijie.sqlite3",
            "include": "illustration,doujin",
            "filename": "{artist_id}.{artist_name}.{image_id}_{title[b:80]}_{num:>02}_{filename[b:130]}.{extension}"
        },

Skipping doesn't seem to work here. The app still spends a lot of time fetching URLs of already downloaded files.

Hrxn commented 2 months ago

Not sure, what does the full log say? I mean, "skip": true means skipping downloads, so I guess this is still working as expected?

reyaz006 commented 2 months ago

From the log I just see that it fetches URL of each item with 2-3 sec pauses, and it takes a long time to go through some individual artists. I don't want to use abort option because I'm still trying to figure out another issue with images that were somehow not downloaded in previous run.

Is there a separate option to skip fetching the URL if the id is in the database?

mikf commented 2 months ago

There isn't. Yet.

You could reduce the wait times with --sleep-request, but that makes it likely to yield 429 errors. Or you might manually adjust the starting pagee number etc in the code to skip over some posts that way: https://github.com/mikf/gallery-dl/blob/4b286e80fd3f591b434f84a2ba415e5d22025f60/gallery_dl/extractor/nijie.py#L154

Gerbilnut commented 1 week ago

I would love to see this enhancement in an upcoming release. I for one would save several hours on each instance if we were able to completely ignore links that were already downloaded without hitting each and every link.