lahwaacz / wiki-scripts

Framework for writing bots, maintenance scripts or performing data analysis on wikis powered by MediaWiki
http://lahwaacz.github.io/wiki-scripts/
GNU General Public License v3.0
27 stars 12 forks source link

Support for cached queries #15

Closed lahwaacz closed 9 years ago

lahwaacz commented 9 years ago

Some scripts fetch big data which is not changing often (or ever), so repeated queries are slow and unnecessarily increase the load on the server. Caching the results of the queries in persistent storage would be solution.

For the storage backend I'm interested in a document-oriented database with easy Python interface. Some considerations:

Some alternatives are mentioned here: http://stackoverflow.com/a/5835099/4180822

lahwaacz commented 9 years ago

I've made some experiments with BlitzDB and either I've missed something important or document databases are not really good for this big data yet. Anyway, custom JSON database seems to have the best effort/performance ratio, so this is being implemented starting with 78ac098260f02eabc0e6e61d014d9b5c1060f23a.