qubole / rubix

Cache File System optimized for columnar formats and object stores
Apache License 2.0
183 stars 74 forks source link

add generation Number #381

Closed suryanshagnihotri closed 4 years ago

suryanshagnihotri commented 4 years ago

Tested the changes by making invalidation happen by :

shubhamtagra commented 4 years ago

Couple of more things:

  1. setAllCached api also needs to be made generationNumber aware because it is possible that it is called after caching data for old gen data file which will cause corruption as currently you are updating the cache status.
  2. We need a test that runs CachedReads in parallel to invalidations being forced