search

navis-org / pymaid

Python library to interface with CATMAID servers. Fully interoperable with navis.
https://pymaid.readthedocs.io/en/latest/
GNU General Public License v3.0
24 stars 11 forks source link

Improvement: write cache to disk #195

Open schlegelp opened 4 years ago

schlegelp commented 4 years ago

Instead of keeping cached responses in memory, we could use a persistent, dictionary-like object saved to disk via e.g. the shelve module.

Need to figure out how this could work with temporary file to make sure it's deleted upon exit.

schlegelp commented 4 years ago

Diskcache seems to be suitable contender.

clbarnes commented 1 year ago

I've been using diskcache a bit recently, seems to work! diskcache is also thread-safe, where I think the current cache isn't (got an exception earlier from what I believe was trying to clear some cache space while iterating over it in another thread).

clbarnes commented 1 year ago

Looping back to this: requests-cache seems ideal. You just wrap your requests.Session with a requests_cache.CachedSession (which can then be wrapped by a requests_futures.FuturesSession), which has various backends including a dict-based one like the existing implementation, and SQLite, which can be in-memory or persist to a file.

By default it only caches code 200 responses, which means we probably wouldn't need cache.undo_on_error any more, and automatically handles expiry etc.. It can also scrub auth headers from the stored requests. It doesn't handle restricting cache size, but that's probably less of a problem if it's writing to disk, and you can set a filter function which would prevent it from caching large data anyway (e.g. images). It may allow cutting down a lot of the current caching code.