search

Serene-Arc / bulk-downloader-for-reddit

Downloads and archives content from reddit
https://pypi.org/project/bdfr
GNU General Public License v3.0
2.31k stars 212 forks source link

RedGifs 401 Errors #679

Closed Soulsuck24 closed 2 years ago

Soulsuck24 commented 2 years ago

Redgifs is broken again.

Now requires client tokens to even access the API. Will need to either get keys to use the API again or switch to how Imgur currently works.

Posting this as a placeholder for anyone else wondering about the issue.

Soulsuck24 commented 2 years ago

I will try to look into it later today to see what can be done to sort it out.

Serene-Arc commented 2 years ago

We have a mechanism in place to pass API keys; it might be time to switch to that and have users register if they want to get RedGif resources. Unless they'd allow a single key for the entirety of the BDFR, but that's only possible if they don't rate-limit based on the key.

Soulsuck24 commented 2 years ago

By the looks of the form you have to fill out they ask what you think your rate limit should be. Doubt would have a way to figure that out though tbh.

Serene-Arc commented 2 years ago

Definitely not, not for the BDFR. It might have to be a per user key, with the BDFR dropping actionless support for RedGifs. I did anticipate the need for user-supplied API keys and there are accommodations for that through the Configuration class, but it hasn't been implemented yet.

Soulsuck24 commented 2 years ago

So, I've been looking into getting this back up. I've gotten a "workaround" that I'm using but it's likely to get nuked not long after pushing to the repo if I was to.

I've been able to get video and single image posts working with the other method but haven't been able to crack albums as of yet.

DanSixtyNine commented 2 years ago

Why is redgifs such a pain in the ass... -.-'

Serene-Arc commented 2 years ago

Sites like this survive on not letting people just download stuff without seeing ads or interacting with the site. Tools like this are anathema to them.

Soulsuck24 commented 2 years ago

My mistyping managed to work out for once... apparently the V1 api is still alive (for now) so while a more permanent solution is figured out this will at least get the Redgifs downloader working again.

Soulsuck24 commented 2 years ago

@Serene-Arc looks like they made more changes (the temp still works for now though).

They look to have added temporary tokens to the API though which seem to be working well (till they break them). They appear to be good for 24 hours so not likely to need more than one per run.

I imagine the site_authenticator would be used for getting a single temporary token per run, but I can't say I've figured out how to code that in myself. If it's not something you have time to do would you be able to point me to some documentation on how it's supposed to work and I can give it a try.

I currently just have it running to the new API with:

auth_token = json.loads(Redgifs.retrieve_url('https://api.redgifs.com/v2/auth/temporary').text)['token']
headers = {
    'Authorization': f'Bearer {auth_token}',
}

content = Redgifs.retrieve_url(f'https://api.redgifs.com/v2/gifs/{redgif_id}', headers=headers)

plus switching the json parsing back to how it is in the currently released version.

Only "downside" is it requests a new token for every request to redgifs. I'd say leave it and let them generate a new token for every request if they want to run their API like that but wouldn't be surprised if they banned BDFR for it even though their webpages literally do that now...

Wolfeman0101 commented 2 years ago

This is still happening. I'm getting 401 errors on RedGifs

Serene-Arc commented 2 years ago

Have you updated to the git version?

Wolfeman0101 commented 2 years ago

I ran pip install bdfr --upgrade. Is there a different way?

Serene-Arc commented 2 years ago

Yes, that only updates the package from PyPi but we haven't done a release with this fix yet. You have to install directly from the development branch of this repository to get the latest version.

germyparker commented 2 years ago

Sites like this survive on not letting people just download stuff without seeing ads or interacting with the site. Tools like this are anathema to them.

So why have an api at all??? it kills me. Instagram is the same way - they have a pretty extensive api, but every time I use it, my account gets nuked.

PS: I'm on the development branch of the git version and getting 401 errors for redgifs. Do I need to make a redgifs account and authenticate thru bdfr?

Serene-Arc commented 2 years ago

You shouldn't have to. Can you post the logs with examples of links that fail so i can check?

germyparker commented 2 years ago

I went to try it just now to generate a fresh log, and it worked without any trouble, so, nevermind, I think? I tinkered with it a lot yesterday - I think I may have been using the pip version despite running it from the bdfr git folder, so I deleted all remnants of the pip version that I could find, and then git pulled. I could've sworn I did the below test after that, but, judging by today's success, I must've done something after it.

So this log may not be useful/accurate, but I'll include it and let you be the judge. 

[2022-11-09 12:23:46,595 - bdfr.connector - DEBUG] - Setting maximum download wait time to 120 seconds
[2022-11-09 12:23:46,595 - bdfr.connector - DEBUG] - Setting datetime format string to ISO
[2022-11-09 12:23:46,596 - bdfr.connector - DEBUG] - Disabling the following modules: 
[2022-11-09 12:23:46,596 - bdfr.connector - Level 9] - Created download filter
[2022-11-09 12:23:46,596 - bdfr.connector - Level 9] - Created time filter
[2022-11-09 12:23:46,596 - bdfr.connector - Level 9] - Created sort filter
[2022-11-09 12:23:46,596 - bdfr.connector - Level 9] - Create file name formatter
[2022-11-09 12:23:46,596 - bdfr.connector - DEBUG] - Using authenticated Reddit instance
[2022-11-09 12:23:46,597 - bdfr.connector - Level 9] - Created site authenticator
[2022-11-09 12:23:46,597 - bdfr.connector - Level 9] - Retrieved subreddits
[2022-11-09 12:23:46,597 - bdfr.connector - Level 9] - Retrieved multireddits
[2022-11-09 12:23:46,598 - bdfr.oauth2 - Level 9] - Loaded OAuth2 token for authoriser
[2022-11-09 12:23:46,936 - bdfr.oauth2 - Level 9] - Written OAuth2 token from authoriser to /Users/-----/Library/Application Support/bdfr/default_config.cfg
[2022-11-09 12:23:47,467 - bdfr.connector - DEBUG] - Retrieving upvoted posts of user ['-----']
[2022-11-09 12:23:47,467 - bdfr.connector - Level 9] - Retrieved user data
[2022-11-09 12:23:47,467 - bdfr.connector - Level 9] - Retrieved submissions for given links
[2022-11-09 12:23:47,801 - bdfr.downloader - DEBUG] - Attempting to download submission rzv7b0
[2022-11-09 12:23:47,802 - bdfr.downloader - DEBUG] - Using Redgifs with url https://redgifs.com/watch/spryknobbygarpike
[2022-11-09 12:23:48,489 - bdfr.downloader - ERROR] - Site Redgifs failed to download submission rzv7b0: Server responded with 401 to https://api.redgifs.com/v2/gifs/spryknobbygarpike
[2022-11-09 12:23:48,490 - root - INFO] - Program complete

Today I received the following log for the same video:

[2022-11-10 09:24:18,347 - bdfr.downloader - DEBUG] - Attempting to download submission rzv7b0
[2022-11-10 09:24:18,347 - bdfr.downloader - DEBUG] - Using Redgifs with url https://redgifs.com/watch/spryknobbygarpike
[2022-11-10 09:24:19,629 - bdfr.downloader - DEBUG] - Written file to /Users/-----/bulk-downloader-for-reddit/output/PublicFlashing/Alicebluwu_PublicFlashing_I love riding the metro cable in my city. great views 😏_rzv7b0.mp4
[2022-11-10 09:24:19,629 - bdfr.downloader - DEBUG] - Hash added to master list: f762654f830b070a6b95945c69ba4c8b
[2022-11-10 09:24:19,629 - bdfr.downloader - INFO] - Downloaded submission rzv7b0 from PublicFlashing

This is just a snip because 1) it was successful and 2) there were like 15 files ahead of it. I can add the rest of the log if you'd like.

Serene-Arc commented 2 years ago

Great, seems to be resolved