Constant Crashes and Finish with 0 Results

[Jmanfrom87]

Getting crashes constantly.

Log File:

{ "levelname": "ERROR", "asctime": "02/05/2024 10:19:51 AM", "filename": "reddit_utils.py", "module": "reddit_utils", "name": "DownloaderForReddit.DownloaderForReddit.utils.reddit_utils", "funcName": "check_authorized_connection", "lineno": 109, "message": "Praw response failed", "exc_info": "Traceback (most recent call last):\n File \"DownloaderForReddit\utils\reddit_utils.py\", line 103, in check_authorized_connection\n File \"praw\util\deprecate_args.py\", line 43, in wrapped\n File \"praw\models\user.py\", line 168, in me\n File \"praw\util\deprecate_args.py\", line 43, in wrapped\n File \"praw\reddit.py\", line 712, in get\n File \"praw\reddit.py\", line 517, in _objectify_request\n File \"praw\util\deprecate_args.py\", line 43, in wrapped\n File \"praw\reddit.py\", line 941, in request\n File \"prawcore\sessions.py\", line 328, in request\n File \"prawcore\sessions.py\", line 264, in _request_with_retries\nprawcore.exceptions.TooManyRequests: received 429 HTTP response" } { "levelname": "ERROR", "asctime": "02/05/2024 10:20:00 AM", "filename": "reddit_utils.py", "module": "reddit_utils", "name": "DownloaderForReddit.DownloaderForReddit.utils.reddit_utils", "funcName": "check_authorized_connection", "lineno": 109, "message": "Praw response failed", "exc_info": "Traceback (most recent call last):\n File \"DownloaderForReddit\utils\reddit_utils.py\", line 103, in check_authorized_connection\n File \"praw\util\deprecate_args.py\", line 43, in wrapped\n File \"praw\models\user.py\", line 168, in me\n File \"praw\util\deprecate_args.py\", line 43, in wrapped\n File \"praw\reddit.py\", line 712, in get\n File \"praw\reddit.py\", line 517, in _objectify_request\n File \"praw\util\deprecate_args.py\", line 43, in wrapped\n File \"praw\reddit.py\", line 941, in request\n File \"prawcore\sessions.py\", line 328, in request\n File \"prawcore\sessions.py\", line 264, in _request_with_retries\nprawcore.exceptions.TooManyRequests: received 429 HTTP response" }

Finish with 0 results

Log:

{ "levelname": "INFO", "asctime": "02/05/2024 10:20:24 AM", "filename": "downloader_for_reddit_gui.py", "module": "downloader_for_reddit_gui", "name": "DownloaderForReddit.DownloaderForReddit.gui.downloader_for_reddit_gui", "funcName": "run", "lineno": 639, "message": "Download thread started" } { "levelname": "INFO", "asctime": "02/05/2024 10:21:37 AM", "filename": "video_merger.py", "module": "video_merger", "name": "DownloaderForReddit.utils.video_merger", "funcName": "merge_videos", "lineno": 90, "message": "Video merger complete", "videos_successfully_merged": 0, "videos_unsuccessfully_merged": 0 } { "levelname": "INFO", "asctime": "02/05/2024 10:21:38 AM", "filename": "download_runner.py", "module": "download_runner", "name": "DownloaderForReddit.DownloaderForReddit.core.download_runner", "funcName": "finish_messages", "lineno": 485, "message": "Download complete", "download_time": "1 min, 13 secs", "significant_user_count": 0, "significant_subreddit_count": 0, "total_user_count": 0, "total_subreddit_count": 0, "post_extraction_count": 0, "comment_extraction_count": 0, "download_count": 0 }

[Jmanfrom87]

Another crash right after I posted this

Log: { "levelname": "INFO", "asctime": "02/05/2024 10:25:39 AM", "filename": "downloader_for_reddit_gui.py", "module": "downloader_for_reddit_gui", "name": "DownloaderForReddit.DownloaderForReddit.gui.downloader_for_reddit_gui", "funcName": "run", "lineno": 639, "message": "Download thread started" } { "levelname": "CRITICAL", "asctime": "02/05/2024 10:26:25 AM", "filename": "main.py", "module": "main", "name": "DownloaderForReddit.main", "funcName": "log_unhandled_exception", "lineno": 48, "message": "Unhandled exception", "exc_info": "Traceback (most recent call last):\n File \"DownloaderForReddit\core\download_runner.py\", line 182, in run\n File \"DownloaderForReddit\core\download_runner.py\", line 240, in run_download\n File \"DownloaderForReddit\core\runner.py\", line 12, in check\n File \"DownloaderForReddit\core\download_runner.py\", line 289, in get_subreddit_submissions\n File \"DownloaderForReddit\core\runner.py\", line 12, in check\n File \"DownloaderForReddit\core\download_runner.py\", line 295, in get_subreddit_submissions\n File \"DownloaderForReddit\core\download_runner.py\", line 298, in handle_submissions\n File \"DownloaderForReddit\core\runner.py\", line 12, in check\n File \"DownloaderForReddit\core\download_runner.py\", line 322, in get_submissions\n File \"praw\models\listing\generator.py\", line 63, in next\n File \"praw\models\listing\generator.py\", line 89, in _next_batch\n File \"praw\util\deprecate_args.py\", line 43, in wrapped\n File \"praw\reddit.py\", line 712, in get\n File \"praw\reddit.py\", line 517, in _objectify_request\n File \"praw\util\deprecate_args.py\", line 43, in wrapped\n File \"praw\reddit.py\", line 941, in request\n File \"prawcore\sessions.py\", line 328, in request\n File \"prawcore\sessions.py\", line 264, in _request_with_retries\nprawcore.exceptions.TooManyRequests: received 429 HTTP response" }

[MalloyDelacroix]

The prawcore 429 - too many requests response is due to reddit's new API limits. There is not currently anything I can do to fix this issue.

[Jmanfrom87]

What about the Redgif ban? I still cant even access it from my browser.

Wouldnt rotation proxy support fix both problems?

EDIT: Nevermind, I see your reply on the other post.

[ericsia]

I thought it was only me facing me this problem. Did reddit have daily limit or monthly? If daily perhaps the application can schedule download limit to prevent crash?

[Jmanfrom87]

I thought it was only me facing me this problem. Did reddit have daily limit or monthly? If daily perhaps the application can schedule download limit to prevent crash?

The main problem for me is Redgifs. I seem to get banned before scraping even one sub. I tried a VPN and configured the program to only use 1 thread and I still got banned before finishing a sub.

The program is great but we really need rotating proxy support. I found people that will program it in for about $500 bucks. I have a post in the discussions tab to see if anybody wants to throw in for it to be built. Ideally, a few people can jump in so just one person doesnt have to pay it all.

[MalloyDelacroix]

@ericsia The problem is reddit does not disclose their API limits. I can't figure out a way to mitigate the restrictions as it always just appears to be random and no information is given in the response.

@Jmanfrom87 I understand you are concerned with the redgifs issue, but there are discussion posts and issues open about that issue. Please try to keep comments under issue discussion about the parent issue.

[puckslane]

I'm also seeing the issue with 429 back offs causing the program to crash and subsequently log out.

I am not great at python, but is it possible to inspect the response header and if it is a 429 sleep further calls for a random period of time between a user defined amount to try and handle the crashing?