mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.7k stars 953 forks source link

Imgur File Naming Error #4025

Closed cheese529 closed 1 year ago

cheese529 commented 1 year ago

I am using this current config to download thousands of imgur links but i've run into an issue. "imgur": { "filename": "{num} - {filename}.{extension}", "directory": ["imgur", "{album['title']} - {album['id']}"], "mp4": true, "postprocessors": [ { "name": "metadata", "filter": "description", "format": "{description}" } ] },

When gallery-dl comes across a non album link (https://imgur.com/Xwo9nlt or https://i.imgur.com/VZC9Qws.jpeg" it places the downloaded image inside a folder with the following variation. "None - None - Imgur" and the file name will be "None - {filename}". How do I change this behavior to make it so that gallery-dl downloads links that are not albums with the following filename {title} - {filename} and then continues to follow the filename I have right now in my config for albums.

Will forever be grateful and appreciate any help on sorting this out asap because of the soon to come imgur purge

mikf commented 1 year ago

Set different filename and directory format strings for different subcategories:

"imgur":
{
    "image": {
        "filename": "{filename}.{extension}",
        "directory": ["imgur"]
    },
    "album": {
        "filename": "{num} - {filename}.{extension}",
        "directory": ["imgur", "{album['title']} - {album['id']}"]
    },
    "mp4": true,
    "postprocessors": [
        {
            "name": "metadata",
            "filter": "description",
            "format": "{description}"
        }
    ]
},

or use :?// to ignore empty fields, our use conditional filenames/directories.

cheese529 commented 1 year ago

Thank you so much Mikf, you are actually the GOAT man. I just got one question, hoping you can explain a bit more or show me on how would I integrate the :?// command into my config assuming I want the filename to also have {title} for single images and many images don't have a title. Also what do you mean by conditional filenames/directories?

use :?// to ignore empty fields, our use conditional filenames/directories.

Thank you lots once again.

mikf commented 1 year ago

Format strings in general are explained here. I guess there should be more examples, but oh well ... Conditional format strings are (somewhat) explained here.

For an optional title: {title:?/_/}

This will be the content of title + _ if title is not empty, or an empty string otherwise.

cheese529 commented 1 year ago

Format strings in general are explained here. I guess there should be more examples, but oh well ... Conditional format strings are (somewhat) explained here.

I tried to read and understand these, even used chatgpt but unfortunately I have zero knowledge about coding or python so it is very difficult for me to understand, that's why I always ask for the exact thing to copy and paste. Thank you for continuing to help me despite my tech skill level. You are an amazing human.

I tried {title:?/_/} like this. But the output on a single file image with no title still looks like this. " - Xwo9nlt" Instead of "Xwo9nlt" "imgur": { "image": { "filename": "{title:?/_/} - {filename}.{extension}", "directory": ["imgur"] }, "album": { "filename": "{num} - {filename}.{extension}", "directory": ["imgur", "{album['title']} - {album['id']}"] }, "mp4": true, "postprocessors": [ { "name": "metadata", "filter": "description", "format": "{description}" } ]

mikf commented 1 year ago

"filename": "{title:?/ - /}{filename}.{extension}"

cheese529 commented 1 year ago

"filename": "{title:?/ - /}{filename}.{extension}"

This works beautifully thank you very much. I am assuming if I want to something similar with "directory": ["imgur", "{album['title']} - {album['id']}"] (remove the - if there is no album title or make gallery-dl write Imgur - if there is no album title, I would do this right? "directory": ["imgur", "{album['title':?/ Imgur - /]} - {album['id']}"]

cheese529 commented 1 year ago

I also tried this but it did not work. "directory": ["imgur", "{album['title']:_/}/Imgur - {album['id']}/"] Please let me know if you know how to do this, @mikf

Hrxn commented 1 year ago

Conditional field is {<name>:?/<prefix>/<suffix>}, so this would be:

"directory": ["imgur", "{album['title']?:/ - /}/Imgur - {album['id']}/"]

But the "Imgur - " part is always present, regardless of{album[title]}` or any other field, not sure if you actually want that.

Anyway, I suggest closing this because the initial question has been answered, and to continue any follow-ups in #3960

cheese529 commented 1 year ago

"Imgur - " part is always present, regardless of{album[title]}` or any other field, not sure if you actually want that.

So is it not possible to make it so that the "Imgur - " part is only present if the album has no title? If that is the case then would it be possible to have the " - " part removed if there is no album title but in case of an album title then "{album['title']} - {album['id']}"

And for sure, I will go ahead and close this now and continue in #3960 . Thank you very much @Hrxn.

Hrxn commented 1 year ago

"Imgur - " part is always present, regardless of{album[title]}` or any other field, not sure if you actually want that.

So is it not possible to make it so that the "Imgur - " part is only present if the album has no title?

No, this should be possible, either use a conditional naming block, or simply something like this:

    "directory": ["imgur", "{album['title']:?/ - /|'Imgur - '}{album['id']}"] 
cheese529 commented 1 year ago
 "directory": ["imgur", "{album['title']:?/ - /|'Imgur - '}{album['id']}"] 

Tried this but it did not work the "Imgur - " part is not present when trying with an album that has no name. https://imgur.com/a/PI0B1fw Instead the output just has the album id.

mikf commented 1 year ago

"directory": ["imgur", "{album['title']|'Imgur'} - {album['id']}"]

cheese529 commented 1 year ago

This fixed it but then it broken again when I introduced a max title code. Am I doing something wrong here?


        "imgur":
        {
            "image": {
                "filename": "{title[200]:?/ - /}{filename}.{extension}",
                "directory": ["imgur"]
            },
            "album": {
                "filename": "{num} - {filename}.{extension}",
                "directory": ["imgur", "{album['title'][200]} - {album['id']}"]
            },
            "mp4": true,
            "postprocessors": [
                {
                    "name": "metadata",
                    "filter": "description",
                    "format": "{description}"
                }
            ]```
mikf commented 1 year ago

I'd be quite helpful for you to actually understand what you're doing, you know, or at least to experiment with this "max title code" on a simple example until you get what is happening and/or why it doesn't work.

Anyway, [200] tells it to grab character number 201 and fails if it does not exist, [:200] or [0:200] returns the first 200 characters (0 to 199) and never fails, even if there are less than 200 characters in the string it is applied to.

AlttiRi commented 1 year ago

Well, less than 1 day is left before NSFW content is removed from Imgur.


I recommend to use "mtime": true in a postprocessor to set modified time to txt files.

I created the follow config:

        "imgur":
        {
            "mp4": true,
            "directory": {
                "subcategory == 'album'": ["[{category}]", "[{category}] {album[date]:%Y.%m.%d}—{album[id]}{album[title][:180]:?—//}"],
                "": ["[{category}]"]
            },
            "filename": {
                "subcategory == 'album'": "[{category}] {num:>03}—{album[id]}—{date:%Y.%m.%d}—{id}—{name}.{extension}",
                "": "[{category}] {date:%Y.%m.%d}—{id}—{name}.{extension}"
            },
            "postprocessors": [
                {
                    "filter": "subcategory == 'album' and description",

                    "name":  "metadata",
                    "mode":  "custom",

                    "filename":  "[{category}] {num:>03}—{album[id]}—{date:%Y.%m.%d}—{id}—{name}.txt",
                    "content-format": "{description}\n",

                    "mtime": true,

                    "archive": "~/gallery-dl/gallery-dl-imgur-postprocessor.sqlite",
                    "archive-prefix": "\fF {category}",
                    "archive-format": "{id}_{hash_sha1(description)[:8]}",
                    "archive-pragma": ["journal_mode=WAL"]
                }
            ]
        },

BTW, I no need to use it anymore?

                {
                    "name": "mtime",
                    "event": "post"
                },

Seems it's enough just to add the "mtime": true, line.


UPD, It's better to use mtime postprocessor. It's required when you use --no-download --no-skip.

cheese529 commented 1 year ago

Anyway, [200] tells it to grab character number 201 and fails if it does not exist, [:200] or [0:200] returns the first 200 characters (0 to 199) and never fails, even if there are less than 200 characters in the string it is applied to.

Thank you for explaining this mikf. I deeply apologize if I am being annoying with all of the issues being opened, I am very new to all of this so I am still trying to learn as we go, this is actually the first command line tool I used, normally I always look for GUI since I do not know how to code but this program is one of a kind, can't find anything else even remotely as good as this. I will continue to make an effort to learn and follow the things you say.