search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.39k stars 930 forks source link

Error when downloading DeviantArt Journal and Status Updates #5916

Open AtomicTEM opened 1 month ago

AtomicTEM commented 1 month ago

[deviantart][error] API responded with 404 Not Found [deviantart][error] Unexpected API response: {'error': 'invalid_request', 'error_description': 'Api endpoint not found.', 'status': 'error'}

Getting error above when trying to download posts/journals, also cannot seem to download status updates whatsoever yet I don't get any error.

hpeter1994 commented 1 month ago

I would like to add that the same error comes up if i want to download anything from my watchlist. It started 2-3 days ago.

[gallery-dl][debug] Version 1.27.1 - Executable (stable/linux)
[gallery-dl][debug] Python 3.10.12 - Linux-6.9.9-arch1-1-x86_64-with-glibc2.39
[gallery-dl][debug] requests 2.31.0 - urllib3 2.2.2
[gallery-dl][debug] Configuration Files ['gallery-dl.conf', 'gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/notifications/watch'
[deviantart][debug] Using DeviantartWatchExtractor for 'https://www.deviantart.com/notifications/watch'
[deviantart][debug] Using custom API credentials (client-id 14xx6)
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/browse/deviantsyouwatch?limit=50&offset=0&mature_content=true HTTP/11" 404 96
[deviantart][debug] {"error":"invalid_request","error_description":"Api endpoint not found.","status":"error"}
[deviantart][error] API responded with 404 Not Found
[deviantart][error] Unexpected API response: {'error': 'invalid_request', 'error_description': 'Api endpoint not found.', 'status': 'error'}
[deviantart][info] No results for https://www.deviantart.com/notifications/watch
mikf commented 1 month ago

The latest API update (July 1st 2024) removed to ton of /browse/… endpoints, some of which were used by gallery-dl (#):

/user/statuses/ is also gone.

There is an alternative for journals and statuses, but I can't find one for deviantsyouwatch or popular.

hpeter1994 commented 1 month ago

By the looks of it deviantart silently restored the /browse/deviantsyouwatch API endpoint, my old script works again.

a-washing-machine commented 5 days ago

Not sure if I should put this here or make a new post for it, but journals seem to be broken again?

It's just that they are broken in a new and creative way I wouldn't have noticed if I didn't do a comparison download on a few test-galleries whenever I switch to a newer gallery-dl version, and noticed a lot of missing images which otherwise would've been linked in journals.

I've tried both gallery-dl 1.27.3 and 1.27.4 just today.

The downloaded files I get are missing the entire text of the journal, they just have some of the formatting and a journal title. Not sure how that happened. On August 20th it was fine using 1.27.3, now it isn't, so it's a site change?

Or is this something on my end? Can somebody else confirm the error?

mikf commented 5 days ago

@a-washing-machine That's a new bug(?) in DA's API. The HTML text content of journals is returned as an empty string:

{
  "html": ""
}
a-washing-machine commented 5 days ago

Well, let's hope deviantArt fixes that themselves, then... -_-

<_mindless ramble that can be ignored_> Ooof, I could imagine some people running gallery-dl during the next few _[unit of time interval until fix]_ and not noticing the error... then have all those empty journals in their download directory they only discover months (or years!) later, mixed in with the rest. After all, the html files being downloaded still have the same file names as the actual journal would, so it wouldn't re-download those particular journals correctly even after gallery-dl was fixed, and of course there would be no easy way to sort out broken journals after the fact. Maybe filtered by file size, I guess. Lucky for me that I caught this upfront, not so lucky for anybody else who doesn't. I sure know that would cause me some frustration. :/ <_/mindless ramble that can be ignored_>