The g3 cache currently does not clear itself under any conditions. This needs attention soon. Certain schemes for clearing the data from the cache come to mind, but many run into different issues when considering different user behavior.
to record the size of each g3 file and the most recent time it was requested. Have an environment variable that sets the max cache usage. When a new query loads a file and the total cache exceeds (say) 90% of the cache limit, free up the oldest items in the cache until you're below threshold.
I like this implementation, and think it should be our first attempt at a properly working cache.
The g3 cache currently does not clear itself under any conditions. This needs attention soon. Certain schemes for clearing the data from the cache come to mind, but many run into different issues when considering different user behavior.
Suggested in https://github.com/simonsobs/sisock/pull/22#pullrequestreview-199674433 is a simple implementation:
I like this implementation, and think it should be our first attempt at a properly working cache.