mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.8k stars 967 forks source link

[deviantart][feature request] For non-image files, is it possible to download BOTH preview image AND artwork? (E.g. for Flash files) #3782

Closed a-washing-machine closed 1 month ago

a-washing-machine commented 1 year ago

For non-image files, I've seen a config option for "download preview image only", and "download actual file", but not both. (extractor.deviantart.original ?)

For non-image file formats such as flash-, .pdf-, .zip-files etc (generally anything that doesn't have a preview in Windows Explorer), I'd like gallery-dl to download both.

Is there a way to do that already, without having to reparse everything twice? (Keep in mind that would disallow use of the abort-parameter.) If not, could this feature be added?

My config:

{
    "extractor":
    {
          "deviantart":
        {
            "cookies": "cookies.txt",
            "refresh-token": "[REDACTED]",
            "client-id": "[REDACTED]",
            "client-secret": "[REDACTED]",          
            "extra": true,
            "metadata": true,
            "blacklist": "foobar",
            "auto-watch": "true",
            "auto-unwatch":"true",

            "postprocessors": [{
                "name": "metadata",
                "mode": "custom",
                "format": "{description}\n"
            }]
        },

    ... ... ...
    }
}
ClosedPort22 commented 1 year ago

(extractor.deviantart.original ?)

That's correct.

Is there a way to do that already

No.

without having to reparse everything twice?

As a workaround, if your filename format has {index} in it, you can write a script (or do it manually) to get a list of IDs of the deviatios you want to re-download. Then use gallery-dl -o original=false https://www.deviantart.com/view/{index} for each deviation ID.

I've had this feature in my personal fork for quite some time and it works pretty well. I decided not to open a PR because it would interfere with --range (enabling this feature can cause a deviation object to yield more than one URL), although this can be fixed by adding an offset option similar to the Tumblr extractor.

a-washing-machine commented 1 year ago

Hmm. Doing this one time to create the needed commands for previously downloaded files might be possible. (We're talking about at least 32.000 non-image files here.)

I mean, it's doable for me make a list of all files of the respective file types, extract the IDs from that, then prepend the string "gallery-dl -o original=false https://www.deviantart.com/view/" to those IDs and run the resulting commands.

But it's not something I can do repeatedly in the future, that just isn't practical. Not to mention it ignores sub-folder / favorites / group folder-organization. :/

ClosedPort22 commented 1 year ago

We're talking about at least 32.000 non-image files here.

Well, that's unfortunate. I'll see if I can get this to work on the original master branch.

a-washing-machine commented 1 year ago

That would be lovely, thanks! :)

ClosedPort22 commented 1 year ago

This feature would work best if #3322 could be merged first.

a-washing-machine commented 1 year ago

This feature hasn't been added yet, has it?

'Cause otherwise my gallery folder would be due for doing a full reparse without abort-parameter to catch those missing preview images in addition to any PDF files gallery-dl failed to download in the prior six months due to a (now corrected) site change.

...Which, without the abort-parameter, is something that would take about a month and a half even without any large number of new images being downloaded, so I'd want to "make it count"! ; )

AlttiRi commented 5 months ago

As I see, based on my downloads, only [".jpg", ".png", ".gif", ".mp4", ".htm"] have no preview. The other extensions like [".zip", ".rar", ".7z", ".swf", ".psd", ".wmv", ...] have a preview image. For them you need to do an extra run with -o original=false:

gallery-dl -o original=false https://www.deviantart.com/view/524929592