The current implementation of MemoryStore(DataStore) is very inefficient. It should be replaced by something that works more like CouchDB but doesn't have the HTTP request overhead.
Documents can be stored in a sorted Z-ordered list, indexed by hashed `%(key)s_%(timestamp)f" strings. Then accessing timestamp ranges reduces to a slice operation.
CouchBase might solve the performance issues, but this (nearly) pure Python option should be maintained as an alternative for offline testing/debugging.
The current implementation of
MemoryStore(DataStore)is very inefficient. It should be replaced by something that works more like CouchDB but doesn't have the HTTP request overhead.Documents can be stored in a sorted Z-ordered list, indexed by hashed `%(key)s_%(timestamp)f" strings. Then accessing timestamp ranges reduces to a slice operation.
We might win a bit by using blist.
CouchBase might solve the performance issues, but this (nearly) pure Python option should be maintained as an alternative for offline testing/debugging.
This might be a start: https://github.com/mastbaum/collie/tree/master/python