Transition from JSON to Database

wlerin / social48-ameblo

Script for archiving entries from ameba blogs

MIT License

3 stars 0 forks source link

Transition from JSON to Database #1

Open wlerin opened 7 years ago

wlerin commented 7 years ago

The NMB48 Official JSON is 411 MB in size, and causes the entire system to hang once the script reaches it. The SKE48 one is not much better, at 209 MB. I might be able to solve this with a more efficient json parsing module (or one that could read/write json in pieces), but switching to a proper (relational or nosql) database would also provide many other benefits, not just on the ameblo side of things but also for the rest of the social48 project.

wlerin commented 7 years ago

TODO

[ ] Research database options and choose a good candidate
[ ] Design schema for all services covered by social48, and try to unify them as much as possible.
[ ] Revise ameblog.py to support the new database
[ ] Import existing json data into the database
[ ] Restore ameblo updating (currently halted due to nmb48official's size) .

wlerin commented 7 years ago

Database Candidates:

~~SQLite~~ (Not much better than JSON)
PostgreSQL
MongoDB
MySQL (Only a consideration because Google Cloud SQL uses it)
Redis

Databases on FullStackPython NoSQL Databases on FullStackPython

Cloud Database Providers: Google Cloud SQL (MySQL) Amazon RDS (PostgreSQL, MySQL, others) BitCan (MySQL, MongoDB)