qubole / rubix

Cache File System optimized for columnar formats and object stores
Apache License 2.0
183 stars 74 forks source link

Reduce the impact of cache pre-warm on the jobs - Part 2 #78

Closed abhishekdas99 closed 6 years ago

abhishekdas99 commented 7 years ago

Parallelize the cache pre warm process by directly reading the data from remote filesystem if block not found in remote node cache. The node where the task is running will read the data remote filesystem as well as will send a download request to the remote node (where the file should be cached). The remote node downloads the file asynchronously.