voussoir / timesearch

The subreddit archiver
BSD 3-Clause "New" or "Revised" License
171 stars 7 forks source link

dont work no more any way around with local dumps?? #20

Closed adikami5665 closed 1 year ago

adikami5665 commented 1 year ago

So i got the local dumps downloaded but I don't know how to proceed and get subreddit file like I used to get in db format. any help to move forward. I know the api is down but can https://github.com/yakabuff/redarc help in any way to generate files like before.

DoaJCBlogger commented 1 year ago

Do you mean that you downloaded the Zstandard files?

adikami5665 commented 1 year ago

yes

voussoir commented 1 year ago

Hi guys. I have added a new module to timesearch that can import the archive files into a timesearch database.

zst is a compressed format. Please use a tool like 7-Zip to extract the actual json files. The files inside the archive do not have a file extension, but that's ok.

Please do a python -m pip install voussoirkit --upgrade just in case I've changed anything in my libraries since you last updated.

Please do a git pull of your timesearch repository to get the new commit 8b3fe524.

Then,

python timesearch.py ingest_jsonfile robotwars_submissions -r robotwars

python timesearch.py ingest_jsonfile robotwars_comments -r robotwars

Hope that helps

image

image

DoaJCBlogger commented 1 year ago

This is great. Thank you. The only thing I would suggest is adding support for Zstandard so you don't have to decompress them.