mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.19k stars 913 forks source link

I want to add a post processor for coomer.party to my gallery.conf file. #2211

Closed left1000 closed 2 years ago

left1000 commented 2 years ago

I intend to add this to the coomer.party section of my conf file

, "postprocessors": [{ "name": "metadata", "mode": "custom", "directory" : "Descriptions", "content-format" : "{content}\n" }

only problem is my conf file does not have a coomer.party section at all, so i'll need to know what the default coomer.party section of the conf file should be so I can add that default section and then amend it with my post processor of choice...

I didn't find this information listed at https://github.com/mikf/gallery-dl/blob/master/docs/gallery-dl.conf nor https://github.com/mikf/gallery-dl/blob/master/docs/gallery-dl-example.conf nor https://github.com/mikf/gallery-dl/blob/master/docs/configuration.rst

also I know coomer is very similar to kemono but the post processor is already working for kemono and that isn't sufficient in the conf file to make it work on coomer.

left1000 commented 2 years ago

Another coomer.party issue... the dates stink. kemono.party's ripper will save the files with date created the date and time you make the gallery-dl command but the date modified will be the date on the kemono.party page which is the date of the update on patreon. that way you get the best two dates.

It seems like the coomer.party functionality of gallery-dl is using the standard date created of whenever gallery-dl makes the download but the date modified seems to be the date the coomer.party importer created the file on their server, not the date referenced in the coomer.party post. Now you might think this is coomer.party's fault, but well, browsing coomer.party in a normal browser you'll that the date that's more useful is included in text on every coomer.party page (just like it appears on kemono.party pages)...

sorry if what I'm saying was poorly explained. The tl;dr of this post and the one above it is both, please make the coomer.party support work as well as the kemono.party support, and these are two minor fringe discrepancies i've noticed.

mikf commented 2 years ago

The category value for coomer.party URLs is coomerparty (instead of kemonoparty for kemono URLs).

You can find these values with -E for any URL:

$ gallery-dl -E https://coomer.party/onlyfans/user/alinity/post/125962203
Category / Subcategory
  "coomerparty" / "onlyfans"
Filename format (default):
  "{id}_{title}_{num:>02}_{filename}.{extension}"
Directory format (default):
  ["{category}", "{service}", "{user}"]
Archive format (default):
  "{service}_{user}_{id}_{num}"

So you can just copy-paste your kemono settings and rename the block to "coomerparty"

Regarding dates, I think kemono also has the wrong date in its Last-Modified headers, especially for older posts. For example https://kemono.party/fanbox/user/3316400/post/109476 was published in 2018, but it gets a 2020 modified date.

You should add an mtime post processor to the list to use the actual published date as modified date:

    "postprocessors": [
        {
            "name": "metadata",
            "mode": "custom",
            "directory" : "Descriptions",
            "content-format" : "{content}\n"
        },
        {
            "name": "mtime"
        }
    ]

Also, kemono and coomer use the exact same code in gallery-dl. Any differences come from the sites and their returned data. gallery-dl handles them both in the same way.

left1000 commented 2 years ago

Okay, so that gets me the text descriptions from coomer.party perfectly. But uh, I don't understand how

{ "name": "mtime" }

is meant to function? I never check the dates on the post processed descriptions, I only ever compare dates on the actual image files. Which as far as I know the post processor doesn't effect? right? it's only for the text files the post processor matters?

or will:

    "coomerparty":
    {
        "archive": "./gallery-dl/cp-archive.sqlite3",
        "cookies": {
            "__ddg1": "",
    "__ddg2": ""
        },
        "postprocessors": [{
            "name": "metadata",
            "mode": "custom",
            "directory"       : "Descriptions",
            "content-format"  : "{content}\n"
        },
        {
        "name": "mtime"
        }]
    },

just magically do everything I asked for? edit: yes, yes it does.

left1000 commented 2 years ago

omfg the name mtime post processor made your example file date modified go from 2020 to 2018!!!

edit:

you're a miracle worker! is there anyway to fix my dates without redownloading the files? my folder is 135gb and 70,000 files and I just learned that some of them have incorrect dates and there's an easy fix :(

left1000 commented 2 years ago

I just found a problem, this postprocessor won't download descriptions for files that already exist, is there anyway around that? A way to find that a file exists as # but still run the post processor to grab descriptions for files I downloaded already in the past? or must I delete the old files to get the post processor to fire?

my guess is no, there's no way to grab those descriptions. It's just it'd be a shame to redownload 30gb of images and video in order to grab 2000 tiny text files :(

Twi-Hard commented 2 years ago

Run it with --no-download --no-skip

left1000 commented 2 years ago

Amazing, thanks, you guys provide such amazing support (even though my questions aren't that smart), asking questions here is better than trying to get help from microsoft or whatever :)