data caching?

[7yl4r]

a nice feature to have would be to allow retention of cached data so that data does not need to be redownloaded.

[ocefpaf]

Based on the size of the data I would image a disk cache, right?

[7yl4r]

Yes, I was imagining:

To save:

• hash the input parameters
• generate filename using {param_hash}_{record_count}.p) tempfile package
• dump results to file w/ pickle package

to fetch before performing query:

• hash input parameters
• check for filenames
• hit the API for record_count only
• load the hash file if unchanged record_count

I'm not certain if record_count is the only parameter to look at, however. Records could be updated without changing the count of records.

[ocefpaf]

I usually go to https://joblib.readthedocs.io/en/latest/auto_examples/memory_basic_usage.html for that kind of operation. If you choose to not save on disk we can use functools.lru_cache from the standard library.