search

qubole / rubix

Cache File System optimized for columnar formats and object stores
Apache License 2.0
183 stars 74 forks source link

fix: usr: #249: Fallback reading from object store if cached file is missing or corrupt #252

Closed abhishekdas99 closed 5 years ago

abhishekdas99 commented 5 years ago

This PR includes:

  1. Fallback reading from object store if cached file is missing or corrupt
  2. Moving core.utils classes (except DummyClusterManager) to common.utils
  3. changed the logic in CacheUtil.getDirectory to work with path object rather than string manipulation. The existing implementation is not going to work for LocalFileSystem (required for unit tests)

The main classes to look at :

  1. Implementation in rubix-core/src/main/java/com/qubole/rubix/core/CachedReadRequestChain.java
  2. Tests in rubix-core/src/test/java/com/qubole/rubix/core/TestCachedReadRequestChain.java
  3. Implementation in rubix-spi/src/main/java/com/qubole/rubix/spi/CacheUtil.java
  4. Tests in rubix-spi/src/test/java/com/qubole/rubix/spi/TestCacheUtil.java
codecov-io commented 5 years ago

Codecov Report

Merging #252 into master will increase coverage by 2.67%. The diff coverage is 60.81%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #252      +/-   ##
==========================================
+ Coverage   47.64%   50.32%   +2.67%     
==========================================
  Files          59       59              
  Lines        2869     2967      +98     
  Branches      308      323      +15     
==========================================
+ Hits         1367     1493     +126     
+ Misses       1421     1387      -34     
- Partials       81       87       +6
Impacted Files Coverage Δ
.../com/qubole/rubix/bookkeeper/HeartbeatService.java 67.27% <ø> (ø) :arrow_up:
...qubole/rubix/common/metrics/BookKeeperMetrics.java 97.29% <ø> (ø) :arrow_up:
.../com/qubole/rubix/core/DirectReadRequestChain.java 67.85% <ø> (+67.85%) :arrow_up:
...m/qubole/rubix/common/utils/DeleteFileVisitor.java 60% <ø> (ø)
...ava/com/qubole/rubix/common/utils/ClusterUtil.java 76.92% <ø> (ø)
.../com/qubole/rubix/presto/PrestoClusterManager.java 62.6% <ø> (ø) :arrow_up:
.../com/qubole/rubix/core/CachingFileSystemStats.java 23.52% <0%> (+23.52%) :arrow_up:
...bole/rubix/bookkeeper/LocalDataTransferServer.java 30% <0%> (-2.24%) :arrow_down:
...java/com/qubole/rubix/core/CachingInputStream.java 0% <0%> (ø) :arrow_up:
...in/java/com/qubole/rubix/common/utils/DataGen.java 0% <0%> (ø)
... and 15 more