search

MalloyDelacroix / DownloaderForReddit

The Downloader for Reddit is a GUI application with some advanced features to extract and download submitted content from reddit.
GNU General Public License v3.0
498 stars 47 forks source link

Sluggish performance the last weeks #221

Closed T-prog3 closed 3 years ago

T-prog3 commented 3 years ago

The performance of the downloader have become painfuly slow the last 2 weeks. I havn't change any settings in forever and still it takes up to 1 hour to download 500 pictures. There is some wierd waiting time where the downloader won't do anything, and then it extracts some more links and start downloading and then go back into sleeping mode a minute. It is so slow right now that i actually can do a much faster job by manualy downloading all the files.

I've tested to tweak the threads and Multi-part settings, start with a fresh build. Tried an old build. But nothing seems to help and there is nothing in the log that gives me any information.

crccheck commented 3 years ago

I added this logging and I noticed I was getting hit with 10+ second sleeps

diff --git a/DownloaderForReddit/local_logging/logger.py b/DownloaderForReddit/local_logging/logger.py
index 6dc0d3b..4100539 100644
--- a/DownloaderForReddit/local_logging/logger.py
+++ b/DownloaderForReddit/local_logging/logger.py
@@ -68,3 +68,12 @@ def make_logger():

     logger.addHandler(stream_handler)
     logger.addHandler(file_handler)
+
+    # See rate limits "message: Sleeping: 10.00 seconds prior to call"
+    def only_sleep(record: logging.LogRecord):
+        return record.msg.startswith('Sleeping: ')
+
+    sleep_logger = logging.getLogger('prawcore')
+    sleep_logger.setLevel(logging.DEBUG)
+    sleep_logger.addFilter(only_sleep)
+    sleep_logger.addHandler(stream_handler)
MalloyDelacroix commented 3 years ago

This is odd. Nothing has changed on the app end that would make this happen. It appears as if reddit is enforcing a stricter rate limit on the app than before. I'm having trouble figuring out why that may be and how to correct it. But I am working on it.

spacewaffles309 commented 3 years ago

Some info here, folks are getting limited to 300 requests per 10 mins unless they use authentication.

https://www.reddit.com/r/redditdev/comments/nj04u4/new_rate_limit_for_json_endpoint_api/

MalloyDelacroix commented 3 years ago

@spacewaffles309 Nice find. Thanks for the info.

I'm hitting the rate limit on the first batch of requests being made when running a download. So I'm assuming they are enforcing the limit by the apps client id instead of something local to the actual user like their ip address.

The great thing about being an installed app was that since it was only issuing get requests, there was no need to authenticate. Looks like we will have to implement a log in system to link the app to users reddit accounts. This was on my roadmap anyway, but I guess I'll have to move it up.