untangle lru_cache and persistent cache

biocommons / hgvs

Python library to parse, format, validate, normalize, and map sequence variants. `pip install hgvs`

https://hgvs.readthedocs.io/

Apache License 2.0

241 stars 94 forks source link

untangle lru_cache and persistent cache #450

Open reece opened 7 years ago

reece commented 7 years ago

The current lru_cache code is complicated because it mixes several intentions. Let's untangle them.

The two goals are 1) in-memory LRU memoization to lessen remote data fetches; 2) persistent caching, primarily so that tests do not require network access.

Consequences of mixing these concerns are:

Can't use external lru_cache code (incl. in 3.x)
Configuration is confusing. uta connect() requires a cache mode that's different than the lru_cache mode, and neither checks whether the supplied value is legit.
Learning mode is slow (probably because it writes back every time).
As implemented, the hdp interface is also entangled in caching.

I'm going to park this as a placeholder for discussion. I think I'd like to see us revert to a cache-unaware UTA module, and a (new) caching hdp that accepts an underlying hdp (eg uta) that merely caches data.

reece commented 6 years ago

Also: Investigate whether we can pin the pickle protocol version to 2 so that the same cache works for Python 2 and 3.

reece commented 6 years ago

Outcomes that I'd like to see:

interface module should be about the interface only. It shouldn't mix caching, etc.
use existing caching and persistence tools where possible
separate caching from the actual data provider (current implementation achieves this)
same cache file for all Python versions
enable caches to be used as sole-source for data
support using NCBI gff3 alignment files directly

Ideas:

Provide distinct LRU (memory) and PersistentCache classes that implement the interface. Use these to provide layers, where nested layers are invoked for cache misses.
Stick to the mix-in model, where sequence fetching, gene data, and transcript data might come from pluggable sources

github-actions[bot] commented 10 months ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.

github-actions[bot] commented 9 months ago

This issue was closed because it has been stalled for 7 days with no activity.

reece commented 8 months ago

This issue was closed by stalebot. It has been reopened to give more time for community review. See biocommons coding guidelines for stale issue and pull request policies. This resurrection is expected to be a one-time event.

github-actions[bot] commented 5 months ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.