search

xavdid / reddit-user-to-sqlite

Pull Reddit user data into a SQLite database
https://pypi.org/project/reddit-user-to-sqlite/
MIT License
217 stars 9 forks source link

This only fetches first 10 pages of resources #5

Closed lxe closed 1 year ago

lxe commented 1 year ago

According to https://github.com/xavdid/reddit-user-to-sqlite/blob/main/reddit_user_to_sqlite/reddit_api.py#L160, it only fetches a subset of your data.

Do I need to modify the code to make it fetch all comments/posts, or is this a limitation of Reddit that this value is hardcoded?

xavdid commented 1 year ago

It's a restriction from Reddit. There's max 10 pages of 100 items each, so there's never a need to read an 11th page.

Instead, use the archive command to ingest your GDPR archive, which lets you fetch and populate the full history.

xavdid commented 1 year ago

I've added info about this to the FAQs.