ashwanthkumar / suuchi

सूचि - Toolkit to build Distributed Data Systems
https://ashwanthkumar.github.io/suuchi/
53 stars 12 forks source link

Sharded store #53

Closed ashwanthkumar closed 7 years ago

ashwanthkumar commented 7 years ago

For some use-cases where we can't have one large database (like RocksDB) for storing multi-terrabyte data. It would be great to have sharding of the Store and a wrapper store that manages it for us automatically.

ShardedStore extends Store, so it's a drop-in replacement for all of the existing stores and you would start having multiple shards automatically. If the existing store already has some data, they might have to be re-played or converted to the newer format. I would have written a tool for migration but since we still don't expose the scan interface on store, it's not possible to write a generic utility that can do this for us.

@brewkode Please check this out.

codecov-io commented 7 years ago

Current coverage is 64.44% (diff: 63.63%)

Merging #53 into master will increase coverage by 0.57%

@@             master        #53   diff @@
==========================================
  Files            26         27     +1   
  Lines           465        481    +16   
  Methods           0          0          
  Messages          0          0          
  Branches         74         80     +6   
==========================================
+ Hits            297        310    +13   
+ Misses          126        123     -3   
- Partials         42         48     +6   

Sunburst

Powered by Codecov. Last update 265d308...7c1e69f

ashwanthkumar commented 7 years ago

Merging this since we need it for our finder benchmarks. Please share any feedback, I'll fix them in a separate PR.