qubole / rubix

Cache File System optimized for columnar formats and object stores
Apache License 2.0
183 stars 74 forks source link

fix: dev: #188: Initialize ClusterManager only from BookKeeper module #194

Closed abhishekdas99 closed 5 years ago

abhishekdas99 commented 6 years ago

The main purpose of this PR is:

  1. ClusterManagers will only get initialized from BookKeeper classes, so that we have it centralized.
  2. The clustermanager has different responsibilities a) Returns the nodes in the cluster (CoordinatorBookkeeper gets this info from the url mentioned in the cluster manager. WorkerBookkeeper gets this info from the coordinator and caches it in local. The main reason for this is to maintain the state of what nodes are live in the cluster now and have centralize place to take care of it.) b) To provide the nodeindex for a particular key using consistent hashing. This needs a key and number of items in the consistent hashing ring. This logic is be invoked by both coordinator as well as worker..
codecov-io commented 6 years ago

Codecov Report

Merging #194 into master will increase coverage by 0.39%. The diff coverage is 45.45%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #194      +/-   ##
==========================================
+ Coverage   50.36%   50.76%   +0.39%     
==========================================
  Files          61       64       +3     
  Lines        3121     3154      +33     
  Branches      348      351       +3     
==========================================
+ Hits         1572     1601      +29     
- Misses       1462     1470       +8     
+ Partials       87       83       -4
Impacted Files Coverage Δ
.../exception/CoordinatorInitializationException.java 25% <ø> (ø)
...a/com/qubole/rubix/bookkeeper/utils/ShellExec.java 75.86% <ø> (ø) :arrow_up:
...a/com/qubole/rubix/bookkeeper/utils/DiskUtils.java 60% <ø> (-0.98%) :arrow_down:
...le/rubix/hadoop2/CachingNativeAzureFileSystem.java 0% <0%> (ø) :arrow_up:
...ubole/rubix/hadoop2/CachingNativeS3FileSystem.java 0% <0%> (ø) :arrow_up:
...ubole/rubix/hadoop2/Hadoop2ClusterManagerUtil.java 61.4% <0%> (-2.24%) :arrow_down:
...qubole/rubix/presto/CachingPrestoS3FileSystem.java 0% <0%> (ø) :arrow_up:
...m/qubole/rubix/core/utils/DummyClusterManager.java 0% <0%> (-13.34%) :arrow_down:
...eeper/exception/WorkerInitializationException.java 0% <0%> (ø)
...le/rubix/hadoop2/CachingDistributedFileSystem.java 0% <0%> (ø) :arrow_up:
... and 26 more
abhishekdas99 commented 6 years ago

@stagraqubole @amoghmargoor Can we review this one ?

abhishekdas99 commented 5 years ago

Ping again for review.