search

csu / export-saved-reddit

Export saved Reddit posts into a HTML file for import into Google Chrome.
Other
433 stars 35 forks source link

Improve performance #32

Closed csu closed 7 years ago

csu commented 7 years ago

Tried running the script today and it was quite slow (running with verbose output on, it looked like it was taking greater than 1 second to process each item). Need identify the bottleneck and fix it. If we're bottlenecked at network or some kind of I/O, we should rewrite the code to be async/parallelized.

tobiasvl commented 7 years ago

Isn't this just because of reddit's API ratelimits?

Clients connecting via OAuth2 may make up to 60 requests per minute. – https://github.com/reddit/reddit/wiki/API

And this script doesn't even use OAuth2, so I assume the limits are even harsher?

csu commented 7 years ago

And this script doesn't even use OAuth2, so I assume the limits are even harsher?

It is using OAuth.

The rate limit is still probably the cause though, so closing this.