bibanon / BASC-Archiver

Python-based Imageboard (4chan) complete thread archiver.
https://pypi.python.org/pypi/BASC-Archiver/
135 stars 18 forks source link

Keep track of JSON better #26

Open DanielOaks opened 8 years ago

DanielOaks commented 8 years ago

We should probably create a JSON manager that loads and saves the thread json file, so we can handle it like *Fuuka does their thread grabbing. For example, being able to put "deleted": true as an attribute of specific posts when they go missing, stuff like that. So we can load and save it as necessary, and keep a more accurate representation of the thread.

antonizoon commented 8 years ago

How about a SQLite database at the root of the archive/4chan/ folder that keeps track of this kind of thing? Simple and elegant, and easily exportable.

DanielOaks commented 8 years ago

Hmm, the issue there is with multiple instances of the archiver running and smashing each others' database handlers/accesses. This mostly refers to the JSON thread manifest files in specific threads' folders though, so we shouldn't need to handle it larger up in the directory tree.