qubole / rubix

Cache File System optimized for columnar formats and object stores
Apache License 2.0
182 stars 74 forks source link

Support file size based usage limits #485

Open lozbrown opened 7 months ago

lozbrown commented 7 months ago

Currently rubix configuration only supports integer percentages for cache limits in the following configuration item

rubix.cache.usage.percentage

However when using an elastic storage like amazon EFS this presents as 8 Exabytes of available storage. Even limiting the cache usage to 1% gives us a 90 Terabyte cache which is extremely excessive.

Support limiting the cache to some number of bytes/megabytes/gigabytes and evict data when either limit is reached.