search

Serene-Arc / bulk-downloader-for-reddit

Downloads and archives content from reddit
https://pypi.org/project/bdfr
GNU General Public License v3.0
2.31k stars 212 forks source link

redgifs video not downloading. Error: Site Redgifs failed to download submission: Could not read the page source [BUG] #335

Closed ymgenesis closed 3 years ago

ymgenesis commented 3 years ago

Description

video doesn't download from redgifs

Command

python3 -m bdfr download [DIRECTORY] --verbose --search-existing --no-dupes --subreddit redgifs --sort top --limit 20 --time all --skip txt

Environment (please complete the following information):

Logs

[2021-05-02 19:56:18,097 - bdfr.downloader - DEBUG] - Setting maximum download wait time to 120 seconds
[2021-05-02 19:56:18,097 - bdfr.downloader - Level 9] - Created download filter
[2021-05-02 19:56:18,098 - bdfr.downloader - Level 9] - Created time filter
[2021-05-02 19:56:18,098 - bdfr.downloader - Level 9] - Created sort filter
[2021-05-02 19:56:18,098 - bdfr.downloader - Level 9] - Create file name formatter
[2021-05-02 19:56:18,098 - bdfr.downloader - DEBUG] - Using unauthenticated Reddit instance
[2021-05-02 19:56:18,099 - bdfr.downloader - INFO] - Calculating hashes for 2 files
[2021-05-02 19:56:18,167 - bdfr.downloader - Level 9] - Created site authenticator
[2021-05-02 19:56:18,168 - bdfr.downloader - DEBUG] - Added submissions from subreddit redgifs
[2021-05-02 19:56:18,168 - bdfr.downloader - Level 9] - Retrieved subreddits
[2021-05-02 19:56:18,168 - bdfr.downloader - Level 9] - Retrieved multireddits
[2021-05-02 19:56:18,168 - bdfr.downloader - Level 9] - Retrieved user data
[2021-05-02 19:56:18,169 - bdfr.downloader - Level 9] - Retrieved submissions for given links
[2021-05-02 19:56:18,818 - bdfr.downloader - DEBUG] - Attempting to download submission gfvw9v
[2021-05-02 19:56:18,820 - bdfr.downloader - DEBUG] - Using Redgifs with url https://redgifs.com/watch/foolishforkedabyssiniancat
**[2021-05-02 19:56:18,984 - bdfr.downloader - ERROR] - Site Redgifs failed to download submission gfvw9v: Could not read the page source**
[2021-05-02 19:56:18,984 - bdfr.downloader - DEBUG] - Attempting to download submission jznkyv
[2021-05-02 19:56:18,984 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/jznkyv/announcement_1_the_future_of_redgifscom/
[2021-05-02 19:56:19,113 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/redgifscom_Announcement #1- The future of RedGIFs.com 🎉_jznkyv.txt
[2021-05-02 19:56:19,114 - bdfr.downloader - DEBUG] - Hash added to master list: 94159a244db22935e8d99bbaf2a8504e
[2021-05-02 19:56:19,114 - bdfr.downloader - INFO] - Downloaded submission jznkyv from redgifs
[2021-05-02 19:56:19,114 - bdfr.downloader - DEBUG] - Attempting to download submission jznyzs
[2021-05-02 19:56:19,114 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/jznyzs/announcement_2_hi_guys_we_cant_wait_to_get_started/
[2021-05-02 19:56:19,216 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/RedGIFsOfficial_Announcement #2- Hi guys, we can’t wait to get started 👋_jznyzs.txt
[2021-05-02 19:56:19,217 - bdfr.downloader - DEBUG] - Hash added to master list: b490877fc227e45dcae717eb9f4dd9d3
[2021-05-02 19:56:19,218 - bdfr.downloader - INFO] - Downloaded submission jznyzs from redgifs
[2021-05-02 19:56:19,218 - bdfr.downloader - DEBUG] - Attempting to download submission ibvmgy
[2021-05-02 19:56:19,218 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/ibvmgy/why_is_redgifs_so_consistently_and_aggravatingly/
[2021-05-02 19:56:19,356 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/HORSEPUSSYENTHUSIAST_Why is redgifs so consistently and aggravatingly slow?_ibvmgy.txt
[2021-05-02 19:56:19,357 - bdfr.downloader - DEBUG] - Hash added to master list: 63632db3f200a25bb8a48c3659d3f664
[2021-05-02 19:56:19,357 - bdfr.downloader - INFO] - Downloaded submission ibvmgy from redgifs
[2021-05-02 19:56:19,357 - bdfr.downloader - DEBUG] - Attempting to download submission gvdyjt
[2021-05-02 19:56:19,357 - bdfr.downloader - DEBUG] - Using Redgifs with url https://redgifs.com/watch/amusedcheerfulirishwaterspaniel
**[2021-05-02 19:56:19,479 - bdfr.downloader - ERROR] - Site Redgifs failed to download submission gvdyjt: Could not read the page source**
[2021-05-02 19:56:19,480 - bdfr.downloader - DEBUG] - Attempting to download submission gzjcml
[2021-05-02 19:56:19,480 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/gzjcml/redgifs_is_so_slow_im_starting_to_feel_like_im/
[2021-05-02 19:56:19,593 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/PadaV4_Redgifs is so slow im starting to feel like im back on dial-up._gzjcml.txt
[2021-05-02 19:56:19,594 - bdfr.downloader - DEBUG] - Hash added to master list: 3061ccd3643019b964f751a416b4d924
[2021-05-02 19:56:19,594 - bdfr.downloader - INFO] - Downloaded submission gzjcml from redgifs
[2021-05-02 19:56:19,595 - bdfr.downloader - DEBUG] - Attempting to download submission jbwlv5
[2021-05-02 19:56:19,595 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/jbwlv5/can_we_get_any_sort_of_meaningful_update_on_the/
[2021-05-02 19:56:19,761 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/SageSteele_Can we get any sort of meaningful update on the general performance issues?_jbwlv5.txt
[2021-05-02 19:56:19,761 - bdfr.downloader - DEBUG] - Hash added to master list: e8fd942ca4e1bde7e20b276256a21263
[2021-05-02 19:56:19,761 - bdfr.downloader - INFO] - Downloaded submission jbwlv5 from redgifs
[2021-05-02 19:56:19,762 - bdfr.downloader - DEBUG] - Attempting to download submission j1batq
[2021-05-02 19:56:19,762 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/j1batq/sooooooo_sloooooow/
[2021-05-02 19:56:19,861 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/mortenmoulder_Sooooooo sloooooow_j1batq.txt
[2021-05-02 19:56:19,861 - bdfr.downloader - DEBUG] - Hash added to master list: a93cc9478986c9b6558e53046a378e85
[2021-05-02 19:56:19,862 - bdfr.downloader - INFO] - Downloaded submission j1batq from redgifs
[2021-05-02 19:56:19,862 - bdfr.downloader - DEBUG] - Attempting to download submission hetc55
[2021-05-02 19:56:19,863 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/hetc55/redgifs_is_an_unacceptably_slow_alternative_at/
[2021-05-02 19:56:19,970 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/ttthrowaway12321_RedGifs is an unacceptably slow alternative at the moment._hetc55.txt
[2021-05-02 19:56:19,970 - bdfr.downloader - DEBUG] - Hash added to master list: ca302c95546f52d5aef0f4d779100e4d
[2021-05-02 19:56:19,970 - bdfr.downloader - INFO] - Downloaded submission hetc55 from redgifs
[2021-05-02 19:56:19,970 - bdfr.downloader - DEBUG] - Attempting to download submission ivw7ww
[2021-05-02 19:56:19,971 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/ivw7ww/is_redgifs_slow_for_anyone_else_or_is_it_just_me/
[2021-05-02 19:56:20,090 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/OnlyUpTooting_Is RedGifs slow for anyone else or is it just me?_ivw7ww.txt
[2021-05-02 19:56:20,090 - bdfr.downloader - DEBUG] - Hash added to master list: 8d3171b74b3538dbeb7da3e73b2d8d54
[2021-05-02 19:56:20,091 - bdfr.downloader - INFO] - Downloaded submission ivw7ww from redgifs
[2021-05-02 19:56:20,091 - bdfr.downloader - DEBUG] - Attempting to download submission hcl8u3
[2021-05-02 19:56:20,091 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/hcl8u3/the_performance_of_redgifs_is_awful_dont_tell_me/
[2021-05-02 19:56:20,193 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/in_case_shit_The performance of Redgifs is awful. Don't tell me it's the same as Gfycat-- it's been a month and I don't think a single Redgif has ever played all the way through on the first try._hcl8u3.txt
[2021-05-02 19:56:20,194 - bdfr.downloader - DEBUG] - Hash added to master list: 682eeba1ef333860e16d1e9f84fbda01
[2021-05-02 19:56:20,194 - bdfr.downloader - INFO] - Downloaded submission hcl8u3 from redgifs
[2021-05-02 19:56:20,194 - bdfr.downloader - DEBUG] - Attempting to download submission gvs1p4
[2021-05-02 19:56:20,194 - bdfr.downloader - DEBUG] - Using Direct with url https://i.redd.it/pcwfco4vho251.png
[2021-05-02 19:56:20,407 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/xTaylorJayx_Redgifs doesn't display GIFs on profile. Instead shows black box instead of loading. Please fix!_gvs1p4.png
[2021-05-02 19:56:20,407 - bdfr.downloader - DEBUG] - Hash added to master list: 66648c49931fd6dabefa8f7732d676da
[2021-05-02 19:56:20,407 - bdfr.downloader - INFO] - Downloaded submission gvs1p4 from redgifs
[2021-05-02 19:56:20,408 - bdfr.downloader - DEBUG] - Attempting to download submission e9fosv
[2021-05-02 19:56:20,408 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/e9fosv/welcome_to_redgifscom/
[2021-05-02 19:56:20,506 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/redgifscom_Welcome to RedGIFs.com!_e9fosv.txt
[2021-05-02 19:56:20,506 - bdfr.downloader - DEBUG] - Hash added to master list: ba4c3a916bc07f7e0dc8cd237d30e4ea
[2021-05-02 19:56:20,506 - bdfr.downloader - INFO] - Downloaded submission e9fosv from redgifs
[2021-05-02 19:56:20,507 - bdfr.downloader - DEBUG] - Attempting to download submission kt2uho
[2021-05-02 19:56:20,507 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/kt2uho/redgifs_official_subreddits_are_here/
[2021-05-02 19:56:20,606 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/RedGIFsOfficial_RedGIFs Official Subreddits are here 🎉_kt2uho.txt
[2021-05-02 19:56:20,606 - bdfr.downloader - DEBUG] - Hash added to master list: 32f762813f242fb2b39c412df63b969e
[2021-05-02 19:56:20,606 - bdfr.downloader - INFO] - Downloaded submission kt2uho from redgifs
[2021-05-02 19:56:20,606 - bdfr.downloader - DEBUG] - Attempting to download submission hj60dn
[2021-05-02 19:56:20,607 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/hj60dn/videos_do_not_appear_or_black_thumbnails/
[2021-05-02 19:56:20,719 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/avgleejav_Videos do not appear or black thumbnails_hj60dn.txt
[2021-05-02 19:56:20,720 - bdfr.downloader - DEBUG] - Hash added to master list: ba5579005531d09dcf1c207ce9040e6b
[2021-05-02 19:56:20,720 - bdfr.downloader - INFO] - Downloaded submission hj60dn from redgifs
[2021-05-02 19:56:20,720 - bdfr.downloader - DEBUG] - Attempting to download submission k9r80x
[2021-05-02 19:56:20,721 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/k9r80x/content_creator_verification_goes_live_today/
[2021-05-02 19:56:20,847 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/RedGIFsOfficial_Content Creator Verification goes live today!_k9r80x.txt
[2021-05-02 19:56:20,848 - bdfr.downloader - DEBUG] - Hash added to master list: 1ccd1edc81e29f18979a7fea6a0499de
[2021-05-02 19:56:20,848 - bdfr.downloader - INFO] - Downloaded submission k9r80x from redgifs
[2021-05-02 19:56:20,848 - bdfr.downloader - DEBUG] - Attempting to download submission kfmyr7
[2021-05-02 19:56:20,849 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/kfmyr7/redgifs_is_now_registered_with_the_cyber_tipline/
[2021-05-02 19:56:20,960 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/RedGIFsOfficial_RedGIFs is now registered with the Cyber Tipline from The National Center for Missing & Exploited Children._kfmyr7.txt
[2021-05-02 19:56:20,960 - bdfr.downloader - DEBUG] - Hash added to master list: b2b225d0724b1264baaecda9fec769f2
[2021-05-02 19:56:20,960 - bdfr.downloader - INFO] - Downloaded submission kfmyr7 from redgifs
[2021-05-02 19:56:20,961 - bdfr.downloader - DEBUG] - Attempting to download submission hkjqyx
[2021-05-02 19:56:20,961 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/hkjqyx/how_to_view_the_blank_redgifs/
[2021-05-02 19:56:21,065 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]redgifs/scoronam_How to view the "blank" RedGifs_hkjqyx.txt
[2021-05-02 19:56:21,065 - bdfr.downloader - DEBUG] - Hash added to master list: 1807203869fa1cd2adc3e58bebc919ea
[2021-05-02 19:56:21,066 - bdfr.downloader - INFO] - Downloaded submission hkjqyx from redgifs
[2021-05-02 19:56:21,066 - bdfr.downloader - DEBUG] - Attempting to download submission j4fkir
[2021-05-02 19:56:21,066 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/j4fkir/any_word_on_the_site_working_properly_for_all/
[2021-05-02 19:56:21,239 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/bubblegrubs_Any word on the site working properly for all users?_j4fkir.txt
[2021-05-02 19:56:21,239 - bdfr.downloader - DEBUG] - Hash added to master list: b824a0b378c7a194cb91ecba436c3736
[2021-05-02 19:56:21,240 - bdfr.downloader - INFO] - Downloaded submission j4fkir from redgifs
[2021-05-02 19:56:21,240 - bdfr.downloader - DEBUG] - Attempting to download submission lw41o3
[2021-05-02 19:56:21,240 - bdfr.downloader - DEBUG] - Using SelfPost with url https://www.reddit.com/r/redgifs/comments/lw41o3/first_step_to_improve_tag_and_search/
[2021-05-02 19:56:21,357 - bdfr.downloader - DEBUG] - Written file to [DIRECTORY]/redgifs/RedGIFsOfficial_First step to improve Tag and Search functionality is now live. We now have a predetermined Tag list. As yet the new tags are not searchable, that functionality is coming soon. Further details provided within post._lw41o3.txt
[2021-05-02 19:56:21,357 - bdfr.downloader - DEBUG] - Hash added to master list: d6c090eb52c0d10602180abf6d36b957
[2021-05-02 19:56:21,357 - bdfr.downloader - INFO] - Downloaded submission lw41o3 from redgifs
[2021-05-02 19:56:21,357 - root - INFO] - Program complete

Also tried multiple other redgif links from various NSFW users on reddit, with no luck. Figured I'd test this error with the redgifs subreddit.

Here's an example attempting to download (NSFW) submitted posts from user aellagirl. Only showing redgifs submissions, all with errors:

[2021-05-02 20:10:30,351 - bdfr.downloader - DEBUG] - Attempting to download submission mi0z6i
[2021-05-02 20:10:30,352 - bdfr.downloader - DEBUG] - Using Redgifs with url https://redgifs.com/watch/lastingpossibleorangutan
[2021-05-02 20:10:30,650 - bdfr.downloader - ERROR] - Site Redgifs failed to download submission mi0z6i: Could not read the page source
[2021-05-02 20:10:30,651 - bdfr.downloader - DEBUG] - Attempting to download submission m7yrd6
[2021-05-02 20:10:30,652 - bdfr.downloader - DEBUG] - Using Redgifs with url https://www.redgifs.com/watch/cleverheartfeltarcticwolf
[2021-05-02 20:10:30,899 - bdfr.downloader - ERROR] - Site Redgifs failed to download submission m7yrd6: Could not read the page source
[2021-05-02 20:10:30,899 - bdfr.downloader - DEBUG] - Attempting to download submission m7lkut
[2021-05-02 20:10:30,899 - bdfr.downloader - DEBUG] - Using Redgifs with url https://www.redgifs.com/watch/mellowfeistyirishredandwhitesetter
[2021-05-02 20:10:31,154 - bdfr.downloader - ERROR] - Site Redgifs failed to download submission m7lkut: Could not read the page source
[2021-05-02 20:10:31,154 - bdfr.downloader - DEBUG] - Attempting to download submission m7e8s6
[2021-05-02 20:10:31,154 - bdfr.downloader - DEBUG] - Using Redgifs with url https://www.redgifs.com/watch/dangerousunselfishgrizzlybear
[2021-05-02 20:10:31,426 - bdfr.downloader - ERROR] - Site Redgifs failed to download submission m7e8s6: Could not read the page source
[2021-05-02 20:10:33,126 - bdfr.downloader - DEBUG] - Attempting to download submission ls6uyo
[2021-05-02 20:10:33,126 - bdfr.downloader - DEBUG] - Using Redgifs with url https://redgifs.com/watch/littlequestionablebactrian
[2021-05-02 20:10:33,412 - bdfr.downloader - ERROR] - Site Redgifs failed to download submission ls6uyo: Could not read the page source

etc.

Serene-Arc commented 3 years ago

This has already been fixed. Please pull from the development branch if you want an immediate fix, else the v2.1.0 release will be coming soon with this fix and others.