ashwanthkumar / suuchi

सूचि - Toolkit to build Distributed Data Systems
https://ashwanthkumar.github.io/suuchi/
53 stars 12 forks source link

Adding first-class support for versioned store #44

Closed brewkode closed 7 years ago

brewkode commented 7 years ago

Added a versioned store that wraps around any suuchi.Store implementation and provides versioning support for keys Differentiates Version records & Actual data records with a prefix so that scans for versions (if need be) can be sped up Explicitly made this project source/target to be java 1.8

@ashwanthkumar Please take a look.

codecov-io commented 7 years ago

Current coverage is 62.92% (diff: 85.45%)

Sunburst

No coverage report found for master at 110d7eb.

Powered by Codecov. Last update 110d7eb...3f39208

ashwanthkumar commented 7 years ago

Ok just now saw the change. I saw that we're adding it to the head earlier too. The ordering maintained here is the insertion order la ji.

I think its not obvious because the version is the timestamp now, one of my earlier comments was make that configurable such that given a key and a value a method has to return a long. And we need that for our use case la. In that scenario to handle out of order updates we need to sort the versions. Appo we will need ordering la.

Text by Ashwanth, typos by Lumia

-----Original Message----- From: "Sriram" notifications@github.com Sent: ‎07-‎10-‎2016 14:49 To: "ashwanthkumar/suuchi" suuchi@noreply.github.com Cc: "Ashwanth Kumar" ashwanthkumar@googlemail.com; "Mention" mention@noreply.github.com Subject: Re: [ashwanthkumar/suuchi] Adding first-class support for versionedstore (#44)

@brewkode commented on this pull request.

In suuchi-core/src/main/scala/in/ashwanthkumar/suuchi/store/VersionedStore.scala:

  • // remove oldest version, if we've exceeded max # of versions per record
  • if(versions.size > numVersions) removeData(key, versions.min) +
  • // Write out the actual data record
  • store.put(dkey(key, versions.max), value) +
  • } +
  • def atomicUpdate(key: Array[Byte], version: Long) = {
  • val versionKey = vkey(key)
  • val absHash = math.abs(MurmurHash3.arrayHash(versionKey))
  • // Synchronizing the version metadata update part alone
  • val monitor = SYNC_SLOTS(absHash % SYNC_SLOTS.length)
  • val versions = monitor.synchronized {
  • val vRecord = store.get(versionKey)
  • val updatedVersions = vRecord.map(bytes => version :: Versions.fromString(new String(bytes)).toList).getOrElse(List(version)) we actually add an item to the head of the list and take only the first 3. Perhaps, that's not clear. I pushed a minor code change in an attempt to make it explicit. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.
ashwanthkumar commented 7 years ago

Ok ignore my previous comment. Wrong context.

ashwanthkumar commented 7 years ago

@brewkode I looked into this branch and make a couple of small changes, rest LGTM. You can merge this anytime.

ashwanthkumar commented 7 years ago

@brewkode Merging this once the build goes green.