mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.87k stars 976 forks source link

404'd gfycat videos should be retried on gifdeliverynetwork/redgifs #874

Closed shinji257 closed 4 years ago

shinji257 commented 4 years ago

NSFW videos were moved to redgifs and some videos went over to gifdeliverynetwork. Redgifs at the very least is supported

Hrxn commented 4 years ago

Are you absolutely positive that they still have the same ID etc.?

shinji257 commented 4 years ago

Yes. Here is an example: http://www.gfycat.com/dimwittedhatefuleasternglasslizard This item produces a 404 when accessing via GFYCat API however via the website redirects it to https://www.gifdeliverynetwork.com/dimwittedhatefuleasternglasslizard

It can be access via RedGIFs at https://www.redgifs.com/watch/dimwittedhatefuleasternglasslizard

This is the same method I've seen elsewhere for handling gfycat videos that were later redirected to be on gdn or redgifs.

shinji257 commented 4 years ago

So I did want to verify (even though I know 100% that the slug is consistent) that doing the redirect would be ok. Found a post.

GfyCat/RedGifs reddit mod saying to workaround current limitation on urls in reddit by replacing redgifs.com/watch with gfycat.com so that they can be embedded. https://www.reddit.com/r/redgifs/comments/e9fosv/welcome_to_redgifscom/

Some subreddits don’t accept redgifs.com links yet. How can I continue posting my GIFs without interruption until some of my subs whitelist redgifs.com?

For the time being, you can use a simple workaround to post in subreddits that don’t accept redgifs.com links for your GIFs. RedGIFs links look like this: [Very NSFW link removed]. You can change “redgifs.com/watch” to “gfycat.com” in a link and it will work like a normal gfycat.com link.

Example NSFW URL that would have been handled as such. This one is hosted over on RedGifs. https://www.redgifs.com/watch/foolishforkedabyssiniancat

If I do their old workaround it would end up as https://www.gfycat.com/foolishforkedabyssiniancat and give a 404 in the downloader. It redirects to GifDeliveryNetwork on the website for that url.

Hrxn commented 4 years ago

TL;DR It works as expected?

shinji257 commented 4 years ago

GfyCat urls are still not being retried in RedGifs extractor. They just generate a 404 url hit right now. Is it possible to do a fallback for it?

AlttiRi commented 2 years ago

@mikf It was already noted here. Some pretty old 404 videos (I suggest they have been not deleted, but lost from API) can be retrieved by a direct URL. But it requires CamelCased ID while URL contains lower cased ID. https://giant.gfycat.com/${CamelCased}, and/or https://thumbs2.redgifs.com/${CamelCased}, -mobile postfix also may help.

I have written this https://github.com/AlttiRi/gfycat-id-camel-caser for my userscript to do that. (Note: I use it with "Iframe API" instead bundling it as a library.)

The minimal logic is trivial (while in my repo I do much more, so it looks not so trivial), but dictionaries (JSON files) may be useful for you.

Hrxn commented 2 years ago

Strange.. I mean, I have no reason to doubt you..

But can you give some example URL(s) where CamelCaseID works while lowecaseID does not?

AlttiRi commented 2 years ago

Feel free to investigate the content. The tool (the site) for CamelCasing is above. luckylivebug limpingcoarsekiskadee illiteratetallirrawaddydolphin freeminorarieltoucan lateunawarebarnswallow Nothing breaking ToS, they just were lost from API, like they (devs) removed the links to webm for all old (before 2021) videos from API response, as well as the title that contained the uploaded file name.

https://thumbs2.redgifs.com/${CamelCased}.mp4 https://thumbs2.redgifs.com/${CamelCased}.webm <- 4 of 5 videos (2nd-5th) above have webm, and it's the original.

WobblyBitterCricket is not available now, while it was downloadable by the direct link in July (It also was 404). Maybe because of it's video of 2021. (Source: https://twitter.com/HydraFXX/status/1414683834382262281)


UPD 2022.03

Sadly, Redgifs have finally purged all webm files even they were the original uploaded files. However, some 404 mp4 are still available by the direct URL.