Smerity / trending_arxiv

Track trending arXiv papers on Twitter from within your circle
MIT License
172 stars 26 forks source link

Move from SQLite to a more concurrent backend #3

Open Smerity opened 8 years ago

Smerity commented 8 years ago

SQLite was fine for personal use but introduces complications when scaling to a multiple user website, even when there is a single write user to many read users. As of now the Twitter refresh that retrieves timeline and search for each followed user results in a pause of a few seconds due to this. The Twitter refresh itself should be improved and more granular but database choice will likely remain a problem regardless.

See SQLAlchemy Pooling:

In the case of SQLite, the SingletonThreadPool or NullPool are selected by the dialect to provide greater compatibility with SQLite’s threading and locking model