search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.7k stars 953 forks source link

[Deviantart] Gallery-dl Create empty Archive when using filter-image in a child-extractor #5242

Closed thanesfauna closed 8 months ago

thanesfauna commented 8 months ago

Hi, I've come across a problem that I can't resolve. I created an image-filter inside the deviantart.stash extractor to ensure that gallery-dl only downloads 'children' stash belonging to the same 'parent' author:

"deviantart":
  {
   "auto-watch": true,
   "parent-metadata": "parent",
   "auto-unwatch": true,
   "extra": true,
   "png": true,
   "include": ["avatar", "background", "scraps", "journal", "status", "gallery"],
   "metadata": true,
   "pagination": "manual",
   "refresh-token": "cache",
   "avatar.formats": ["original.jpg", "big.jpg", "big.gif", ".png"],
   "quality": "png",
   "sleep-request": 2,
   "stash":
    {
        "image-filter":  "'parent' in locals() and parent['username'] is author['username']",
    },
   "filename": "{title!C} _{index_base36|index}.{extension}",
   "directory": {
       "": ["Deviantart", "{username}", "Gallery", "{da_category}"]
},
   "archive": */Deviantart/{username}.sqlite3",
   "archive-format": " {subcategory:?//} {index_base36|index} {date} {extension}",

however, even though it works and gallery-dl does not download these images, it still creates an empty archive file that has the name of the skipped author["username"] (in this case an author called "ThadyGrant") here is the link that creates this problem: (NSFW i think?} https://www.deviantart.com/cedar-ashens/journal/Lirus-Project-CLOSED-539840139 Is there a way to prevent this from happening?

mikf commented 8 months ago

No, that's not possible. Archive initialization (and creation if it does not exist) happens before image-filter and there is no other filter etc to prevent it being created.

Why not use one single archive file for everything DA?

parent['username'] is author['username']

Use == to check if two strings are equal, not is. is is not guaranteed to work, even if both sides contain the same value.

>>> a = "test"
>>> b = a.lower()
>>> a == b
True
>>> a is b
False

"avatar.formats": ["original.jpg", "big.jpg", "big.gif", ".png"],

"avatar": {
    "formats": ["original.jpg", "big.jpg", "big.gif", ".png"]
},
thanesfauna commented 8 months ago

Thank you very much for helping!

Twi-Hard commented 8 months ago

Couldn't a postprocesser be used to remove it shortly after it's made? It could remove the file if it meets a certain condition if needed?