mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.76k stars 964 forks source link

[Question] gallery-dl ignoring config for imgur when downloading subreddit #4014

Open TheCrimsonLady opened 1 year ago

TheCrimsonLady commented 1 year ago

Hi,

I've combed through quite some issues here now but I just can't seem to find my mistake. I just want to download a subreddit, where all images are saved with a folder that is named after the subreddit. Current example: MaidsOnSkates for some reason, with my config, gallery-dl makes another directory next to "reddit" instead of saving the images in the directory. ideally I'd like this structure:

reddit/MaidsOnSkates/images.ext
          /ImaginarySliceOfLife/images.ext
          /ImaginaryCyberpunk/images.ext
...

I'd be very grateful for any advice.

My config:

{"imgur":
{
    "filename": {
        "'_reddit' in locals()": "{_reddit[title]} {_reddit[date]:%Y-%m-%d} {_reddit[id]}{num:?-//}.{extension}"
    },
    "mp4": true,
    "postprocessors": [
        {
            "name": "metadata",
            "filter": "description",
            "format": "{description}"
        }
    ]
},
"reddit":
{
    "subreddit":
    {
        "directory": ["Reddit", "Subreddits", "{subreddit}"],
        "filename": "{title[:180]!t:?/_/R /_/R&/and/}{id}{num:?_//>02}.{extension}"
    },
    "user":
    {
        "directory": ["Reddit", "Users", "{author}"],
        "filename": "{title[:180]!t:?/_/R /_/R&/and/}{id}{num:?_//>02}.{extension}"
    },
    "archive": "/path/to/subreddit-archive.log",
    "parent-directory": true,
    "parent-metadata": "_reddit_",
    "client-id": "gallery-dl",
    "refresh-token": "TOKEN",
    "postprocessors": ["nonpicsort"]
}
}

command:

/usr/local/bin/gallery-dl --config /path/to/gallery-dl-config/custom-conf.json  -I /path/to/subreddits/list.txt --download-archive /path/to/new-conf/new-reddit-archive.log -d /path/to/subreddits/new-conf/ --sleep-request 3

to reproduce the issue, simply add these links to list.txt:

https://www.reddit.com/r/maidsonskates
https://www.reddit.com/r/ImaginarySliceOfLife/
https://www.reddit.com/r/ImaginaryCyberpunk/
mikf commented 1 year ago

Is this your entire config? I am asking because imgur and reddit need to be placed inside an extractor block for them to have any effect.

You are also setting parent-metadata to _reddit_., but are checking for _reddit without underscore at the end.

Try this one ```` json { "extractor": { "imgur": { "filename": { "'_reddit' in locals()": "{_reddit[title]} {_reddit[date]:%Y-%m-%d} {_reddit[id]}{num:?-//}.{extension}" }, "mp4": true, "postprocessors": [ { "name": "metadata", "filter": "description", "format": "{description}" } ] }, "reddit": { "subreddit": { "directory": ["Reddit", "Subreddits", "{subreddit}"], "filename": "{title[:180]!t:?/_/R /_/R&/and/}{id}{num:?_//>02}.{extension}" }, "user": { "directory": ["Reddit", "Users", "{author}"], "filename": "{title[:180]!t:?/_/R /_/R&/and/}{id}{num:?_//>02}.{extension}" }, "archive": "/path/to/subreddit-archive.log", "parent-directory": true, "parent-metadata": "_reddit", "postprocessors": ["nonpicsort"] } } } ````
TheCrimsonLady commented 1 year ago

ah well yeah that would explain a lot lol

at some point I had an extractor block but between edits it must've gotten lost and the config more and more broken

if I wanted to add all images flat into the subreddit folder, what would I have to remove?

and if there are more sites like deviant art where the actual pictures are, can I just replicate the imgur section with just the site name instead of "imgur"?

mikf commented 1 year ago

Take a look at https://github.com/mikf/gallery-dl/discussions/3958. Set directory to an empty value for all extractors and only give it a real value for reddit.

can I just replicate the imgur section with just the site name instead of "imgur"?

That would work as well, but, depending on the number of different external sites, this could be quite a chore.

You can find the site names like "imgur" with -E, by the way.

$ gallery-dl -E https://www.deviantart.com/shimoda7/art/For-the-sake-10073852
Category / Subcategory
  "deviantart" / "deviation"