mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.7k stars 953 forks source link

gallery-dl.conf? Way to disable site name folder injection into folder paths? #1841

Closed earthyrock closed 3 years ago

earthyrock commented 3 years ago

My explanation: I create a number of folders that each have the artist names, as shown on deviantart.com. EACH folder has a batch file that runs this:

@echo off setlocal EnableExtensions DisableDelayedExpansion pushd "%~dp0" || exit /B move /Y "Folder2.jpg" "Folder.jpg" for %%I in (.) do set "FOLDER=%%~nxI" "%ProgramFiles(x86)%\gallery-dl\gallery-dl.exe" -d "X:\11Web\gallery-dl" --download-archive "%~dp0zzzGDB.sqlite3" "https://www.deviantart.com/%FOLDER%/gallery/all" if not errorlevel 1 if exist "zzzGDB.sqlite3" del "Folder.jpg" popd endlocal

I create a number of folders that each have the artist numbers, as shown on pixiv.com. EACH folder has a batch file that runs this:

@echo off setlocal EnableExtensions DisableDelayedExpansion pushd "%~dp0" || exit /B move /Y "Folder2.jpg" "Folder.jpg" for %%I in (.) do set "FOLDER=%%~nxI" for /f "tokens=1 delims= " %%a in ("%FOLDER%") do set tmpFOLDER=%%a rem if not exist "zzzGDB.sqlite3" "%ProgramFiles(x86)%\gallery-dl\gallery-dl.exe" -d "X:\11Web\gallery-dl" --download-archive "%~dp0zzzGDB.sqlite3" "https://www.pixiv.net/en/users/%tmpFOLDER%/illustrations" if exist "zzzGDB.sqlite3" del "Folder.jpg" if not exist "zzzGDB.sqlite3" del "Folder3.jpg" if exist "zzzGDB.sqlite3" move /Y "Folder3.jpg" "Folder.jpg" if not errorlevel 1 if exist "zzzGDB.sqlite3" del "Folder.jpg" TIMEOUT /T 1 popd endlocal

When I run either of the above from within the "Gallery-DL" folder that I have as root, gallery-dl appears to have some functionality that injects a sub-folder with the name of the respective site I am downloading from. If I am downloading from deviantart, it will inject a DeviantArt subfolder within the root of Gallery-DL, and then download my files in the artist name folder.

If I am downloading from pixiv, it will inject a Pixiv subfolder within the root of Gallery-DL, and then download my files in the artist number folder.

Desired X:\11Web\gallery-dl\romandubina\ X:\11Web\gallery-dl\64090492

Not Desired X:\11Web\gallery-dl\DeviantArt\romandubina\ X:\11Web\gallery-dl\Pixiv\64090492

This appears to be some internal functionality of the gallery-dl app, and/or I cannot find where same is managed/modified within the gallery-dl.conf file.

Thanks so much!!

mikf commented 3 years ago

You'll have to modify the directory setting. (-d only affects base-directory)

By default it includes "{category}" as its first item, which translates to deviantart or pixiv, but you seem to have already changed them yourself (DeviantArt, Pixiv).

earthyrock commented 3 years ago

Sorry, was "Deviantart" not "DeviantArt"


The first 3rd of the gallery-dl.conf:

{ "extractor": { "base-directory": "./gallery-dl/", "postprocessors": null, "archive": null, "cookies": null, "cookies-update": false, "proxy": null, "skip": true, "sleep": 0, "path-restrict": "auto", "path-replace": "_", "path-remove": "\u0000-\u001f\u007f", "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36", "postprocessors": [ { "name": "ugoira", "whitelist": ["pixiv", "danbooru"], "extension": "mp4", "keep-files": false, "ffmpeg-twopass": false, "ffmpeg-args": ["-hide_banner", "-loglevel", "quiet", "-c:v", "libx264", "-b:v", "2M", "-preset", "veryfast", "-pix_fmt", "yuv420p"], "ffmpeg-location": "C:\Program Files\ffmpeg\bin\ffmpeg.exe" }, { "name": "metadata", "whitelist": ["danbooru", "yandere", "sankaku"], "mode": "tags" } ],


So, unbeknownst to me, I did indeed have a "directory" line @ Pixiv section:

    "pixiv":
    {
        "directory": ["Pixiv", "{user[id]}"],
        "username": "*********************",
        "password": "*********************",
        "avatar": false,
        "ugoira": true
    },

Removing "Pixiv", did the trick for Pixiv ;) Thanks. "directory": ["{user[id]}"],

Worked perfect, output is as desired: X:\11Web\gallery-dl\64090492


As for DeviantArt, I had no such line:

    "deviantart":
    {
        "extra": false,
        "client-id": "*****",
        "client-secret": "**********************************************",
        "flat": true,
        "folders": false,
        "journals": "html",
        "mature": true,
        "metadata": false,
        "original": true,
        "quality": 100,
        "refresh-token": "cache",
        "wait-min": 0,
        "cookies": "C:\\Program Files (x86)\\gallery-dl\\cookies.txt",
        "cookies-update": true
    },

Added the line you suggested to the Deviantart section above. That resolved the folder injection issue, w/ the caveat, below.

Added:

        "directory": ["{user[id]}"],

New issue, does Deviantart user have something other than {user[id]} or {user[name]}? When running either of the below, target folders now resolve to "none" (as I'm viewing the ongoing download in cmd prompt, or afterward looking in Windows Explorer.).

        "directory": ["{user[id]}"],

Resultant folder named "none." X:\11Web\gallery-dl\none

        "directory": ["{user[name]}"],

Resultant folder named "none." X:\11Web\gallery-dl\none

Please, what is the correct syntax to capture the Deviantart id/user in order that the final target folder has the artist's name?

Example:

X:\11Web\gallery-dl\romandubina


The final 3rd of the gallery-dl.conf:

"downloader":
{
    "part": true,
    "part-directory": null,

    "http":
    {
        "adjust-extensions": true,
        "mtime": true,
        "rate": null,
        "retries": 4,
        "timeout": 30.0,
        "verify": true
    },

    "ytdl":
    {
        "format": null,
        "forward-cookies": false,
        "mtime": true,
        "outtmpl": null,
        "rate": null,
        "retries": 4,
        "timeout": 30.0,
        "verify": true
    }
},

"output":
{
    "mode": "auto",
    "progress": true,
    "shorten": true,
    "log": "[{name}][{levelname}] {message}",
    "logfile": null,
    "unsupportedfile": null
},

"netrc": false

}}


mikf commented 3 years ago

Please, what is the correct syntax to capture the Deviantart id/user in order that the final target folder has the artist's name?

author[username] You can see all available field names with gallery-dl -K URL.

pixiv doesn't need a password anymore, by the way.

earthyrock commented 3 years ago

"gallery-dl -K URL" worked great. clap clap clap clap clap clap

Thanks so much for taking time out to work with me on this. Gonna call this resolved. . clap clap

HAVE A GREAT WEEK!!