twitter-archive / torch-dataset

An extensible and high performance method of reading, sampling and processing data for Torch
Apache License 2.0
76 stars 24 forks source link

Cache needs a proper eviction implementation. #1

Closed zakattacktwitter closed 8 years ago

zakattacktwitter commented 8 years ago

By default we do not evict anything from the SlowFS cache. When evict is enabled we are very aggressive, removing files from the cache as soon as they are marked for possible eviction.

We need a proper thread/process safe cache eviction implementation. It should only remove files when the cache needs more space for a new file. It should be provided a file system size limit on creation.