search

xavdid / reddit-user-to-sqlite

Pull Reddit user data into a SQLite database
https://pypi.org/project/reddit-user-to-sqlite/
MIT License
215 stars 9 forks source link

Fix HTTP 429 throttling errors: add API backoff and rate limiting. #24

Closed piyh closed 1 year ago

piyh commented 1 year ago

Addresses issue https://github.com/xavdid/reddit-user-to-sqlite/issues/23. I was getting 429 errors, after this PR I was able to download all 16k of my comments without fail.

This PR adds rate limiting on API calls to get comments, with delays between requests for batches more than 10k comments, and also adds retries with exponential backoff for general errors.

I'd like to celebrate my first open source PR with an emoji 🎉

xavdid commented 1 year ago

This is awesome, I appreciate the contribution!

I'll try out your changes this week and then merge them.

There's a couple of things I'm going to tweak before release. The big thing is that we'll want to add some unit tests to make sure this all works like we expect. You're welcome to take a crack at that, or I can handle it on my end. There's also some minor style things to fix (using snake_case variable names instead of camelCase, for instance) but without CI or a linter, you'd have no way of knowing you needed to do that.

I'm glad you were able to get your data exported cleanly 😁

brandongalbraith commented 1 year ago

Thank you @piyh for implementing this! 🙇 👏