mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.84k stars 974 forks source link

[kemono.party/coomer.party] API endpoint changed #4676

Closed sanddudu closed 1 year ago

sanddudu commented 1 year ago

kemonoparty: HttpError: '404 NOT FOUND' for 'https://kemono.party/api/patreon/user/***'

https://kemono.party/api -> https://kemono.party/api/v1

sanddudu commented 1 year ago

schema: https://kemono.party/api/swagger_schema

outohi commented 1 year ago

thank you my friend!

owen135246 commented 1 year ago

Sorry, how should I handle new download commands? At present, I am unable to run the download command and will display this

C:\Users\owen>gallery-dl "https://kemono.party/patreon/user/4707259" -d "E:/gallery-dl/"
[kemonoparty][error] HttpError: '404 NOT FOUND' for 'https://kemono.party/api/patreon/user/4707259'
asphyxiaxx commented 1 year ago

Sorry, how should I handle new download commands? At present, I am unable to run the download command and will display this

C:\Users\owen>gallery-dl "https://kemono.party/patreon/user/4707259" -d "E:/gallery-dl/"
[kemonoparty][error] HttpError: '404 NOT FOUND' for 'https://kemono.party/api/patreon/user/4707259'

Go to gallery-dl install location in your pc, find kemonoparty.py and edit it, replace all api/ to api/v1/

left1000 commented 1 year ago

Any chance we could get a release of 1.26.1 for this? I'm not sure I'm skilled enough to make gallery-dl for myself. I've always relied on the .exe downloads.

sanddudu commented 1 year ago

@owen135246 as my PR is not accepted at the moment, you could manually replace kemonoparty.py with my version. https://github.com/sanddudu/gallery-dl/blob/6b22af9720f4599eb6e2cb524f6f7aa57846ad17/gallery_dl/extractor/kemonoparty.py

owen135246 commented 1 year ago

@owen135246由于我的 PR 目前尚未被接受,您可以手动将 kemonoparty.py 替换为我的版本。 https://github.com/sanddudu/gallery-dl/blob/6b22af9720f4599eb6e2cb524f6f7aa57846ad17/gallery_dl/extractor/kemonoparty.py

Thank you very much. I have replaced the "kemonoparty. py" file with your help. How should I modify my download command?

sanddudu commented 1 year ago

@owen135246 Just use everything as usual.

owen135246 commented 1 year ago

Sorry, how should I handle new download commands? At present, I am unable to run the download command and will display this

C:\Users\owen>gallery-dl "https://kemono.party/patreon/user/4707259" -d "E:/gallery-dl/"
[kemonoparty][error] HttpError: '404 NOT FOUND' for 'https://kemono.party/api/patreon/user/4707259'

Go to gallery-dl install location in your pc, find kemonoparty.py and edit it, replace all api/ to api/v1/

Is it the 15kb file in 'gallery-dl-master\gallery_dl\extractor\kemonoparty.py' that needs to be replaced? I have replaced all 'apis/' inside with 'apis/v1/'. But using the original instructions will still fail.

C:\Users\owen>gallery-dl "https://kemono.party/patreon/user/3223674" -d "E:/gallery-dl/"
[kemonoparty][error] HttpError: '404 NOT FOUND' for 'https://kemono.party/api/patreon/user/3223674'
owen135246 commented 1 year ago

@owen135246 Just use everything as usual.

Excuse me, do I need to replace the 15kb file in 'gallery-dl-master\gallery_dl\extractor\kemonoparty.py'? I have replaced it, but using the original 'gallery-dl "https://kemono.party/patreon/user/3223674" -d "E:/gallery-dl/"' command will still display '[kemonoparty][error] HttpError: '404 NOT FOUND' for 'https://kemono.party/api/patreon/user/3223674''.

C:\Users\owen>gallery-dl "https://kemono.party/patreon/user/3223674" -d "E:/gallery-dl/"
[kemonoparty][error] HttpError: '404 NOT FOUND' for 'https://kemono.party/api/patreon/user/3223674'
sanddudu commented 1 year ago
  1. This thread is only meant for discussion of the problem of API changes, not a help desk.
  2. Replacing the extractor code only works when running gallery-dl from the source code or any package manager that saves source code locally, if you are running the executable please wait for the new version.

The path of the code: <ROOT_OF_SOURCE>\gallery_dl\extractor\kemonoparty.py

owen135246 commented 1 year ago
  1. This thread is only meant for discussion of the problem of API changes, not a help desk.
  2. Replacing the extractor code only works when running gallery-dl from the source code or any package manager that saves source code locally, if you are running the executable please wait for the new version.

The path of the code: <ROOT_OF_SOURCE>\gallery_dl\extractor\kemonoparty.py

Okay, thank you for your information and help. I look forward to your next update.

mikf commented 1 year ago

4677 is merged.

Updated executables can be found at https://github.com/mikf/gallery-dl/actions/runs/6576981807

Any chance we could get a release of 1.26.1 for this?

This weekend.

asphyxiaxx commented 1 year ago

A question here, the filename with the format {date:%Y-%m-%d} is no longer functioning but using {date} will work as expected. Prior to the API change, the format {date:%Y-%m-%d} was functional."

Following is the complete code I were used: gallery-dl https://kemono.party/fanbox/user/12345 -f "{date:%Y-%m-%d} {title} {num}_{filename[:180]}.{extension}" Output: [kemonoparty][error] FilenameFormatError: Applying filename format string failed (ValueError: Invalid format specifier '%Y-%m-%d' for object of type 'str')

mikf commented 1 year ago

{date} and various other smaller things should be fixed in https://github.com/mikf/gallery-dl/commit/c9a976d8a62c0e90abcee2d09dafc8a43fea543b. I'm not sure if discord channels will return everything, since they don't appear to be paginated anymore. (https://github.com/mikf/gallery-dl/commit/174191cb79f1d22c095d499c7fadbf7ff0587da0)

biggestsonicfan commented 1 year ago

@mikf Looks like there's a rare date exception when the post is published on the exact second.

mikf commented 1 year ago

@mikf Looks like there's a rare date exception when the post is published on the exact second.

I'm sorry, but I don't see anything wrong with the results returned by gallery-dl for this post. {date} is a perfectly valid datetime object.

biggestsonicfan commented 1 year ago

@mikf Looks like there's a rare date exception when the post is published on the exact second.

I'm sorry, but I don't see anything wrong with the results returned by gallery-dl for this post. {date} is a perfectly valid datetime object.

Log when downloading via the user:

kemonoparty: An unexpected error occurred: ValueError - Invalid format specifier '%Y-%m-%d' for object of type 'str'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
kemonoparty: 
Traceback (most recent call last):
  File "/home/rob/.local/lib/python3.11/site-packages/gallery_dl/job.py", line 125, in run
    self.dispatch(msg)
  File "/home/rob/.local/lib/python3.11/site-packages/gallery_dl/job.py", line 172, in dispatch
    self.handle_directory(msg[1])
  File "/home/rob/.local/lib/python3.11/site-packages/gallery_dl/job.py", line 350, in handle_directory
    callback(self.pathfmt)
  File "/home/rob/.local/lib/python3.11/site-packages/gallery_dl/job.py", line 609, in _call_hook
    callback(pathfmt)
  File "/home/rob/.local/lib/python3.11/site-packages/gallery_dl/postprocessor/metadata.py", line 98, in run
    path = directory + self._filename(pathfmt)
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rob/.local/lib/python3.11/site-packages/gallery_dl/postprocessor/metadata.py", line 155, in _filename_custom
    self._filename_fmt(pathfmt.kwdict)))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rob/.local/lib/python3.11/site-packages/gallery_dl/formatter.py", line 129, in format_map
    result[index] = func(kwdict)
                    ^^^^^^^^^^^^
  File "/home/rob/.local/lib/python3.11/site-packages/gallery_dl/formatter.py", line 172, in wrap
    return fmt(kwdict[key] if key in kwdict else self.default)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rob/.local/lib/python3.11/site-packages/gallery_dl/formatter.py", line 409, in wrap
    return format(obj, format_spec)
           ^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Invalid format specifier '%Y-%m-%d' for object of type 'str'

Ah the issue is with formatting for filenames, not the download itself. My metadata's filenames are "filename": "{date:%Y-%m-%d}_{id}_{title}.json",. I don't even use this postprocessor anymore so I can bypass the issue, but it's strange the date cannot be parsed when all other posts worked fine for that processor.

mikf commented 1 year ago

@biggestsonicfan fixed in https://github.com/mikf/gallery-dl/commit/3bbaa875f1749bb06a3a2d47fad07d0dbd48e911. It was the next post that had a non-standard date format.