Feature/replication api

[telamon]

Hey! This is the second feature I have in stock for multifeed (mentioned it earlier in Issue #2 )

It formalizes and exposes the feed-exchange to applications in order to let an application attach it's own metadata and control which feeds it want's to share and receive.

Example policies:

var StaleFeedPolicy = {
  have: function(local, share) {
    share(local.keys, {
      lastActivity: extractLastEntryTimestamp(local.keys) // => [Date, Date, Date]
    })
  },
  want: function(remote, request) {
    var keys = remote.keys.filter(function(key, i){
      return remote.lastActivity[i] < 1000*60*60*24*7 // Accept if within 7 days
    })
    request(keys)
  }
}

var FeedTypePolicy = {
  have: function(local, share) {
    share(local.keys, {
      feedTypes: extractTypeFromFeeds(local.keys), // => ['chat', 'images', 'chat']
      sfw: extractSafeFlag(local.keys) // => [true, true, false]
    })
  },
  want: function(remote, request) {
    var keys = remote.keys.filter(function(key, i){
      // Accept only safe-for work chat feeds.
      return remote.feedTypes[i] == 'chat' && remote.sfw[i]
    })
    request(keys)
  }
}

multifeed.use(StaleFeedPolicy)
multifeed.use(FeedTypePolicy)
// Now, multifeed only accepts safe for work chat feeds that has entries
// within the past week.
// But all feeds are still available for replication to peers with different preferences.

I'm a little short on words today, hope the above examples explains the intention of the feature. Will gladly receive feedback on the policy/adapter design as this was thrown together as a proof of concept.

Maybe it's more useful to split it up as 2 separate types of plugins? multifeed.useShareProvider(...) and multifeed.useReceiveFilter(...)

[okdistribute]

Amazing work! Totally readable and understandable with nice tests. Awesome.

[telamon]

So while writing the tests yesterday I couldn't quite let go of the feel of stiffness when I wanted to expose some info on the meta-exchange that required looping through async callbacks.

So I prototyped some syntatic sugar for the policies. It's entirely possible to provide both styles at the same time, but I'm unsure if it's worth implementing unless it provides better readability and ease of use:

Given an multifeed instance:

  var multi = multifeed(hypercore, ram, { valueEncoding: 'json' })

Attach an announcement meta-info extender (ugh.. terminology..):

  multi.useShareProvider(function(feed, share, ignore) {
    feed.get(0, function(err, entry) {
      if (err) ignore()
      else share({timestamp: entry.createdAt})
    })
  })

Attach an inline feed accept filter:

  multi.useAcceptFilter(function(feedMeta, accept) {
    accept(new Date().getTime() - feedMeta.timestamp < 86400000)
  })

The benefit of this pattern/style is that the application does not have to handle lists of feeds, which might also be considered a limitation by some. Thoughts?

[m4gpi]

This is ace and exciting @telamon :raised_hands:

[m4gpi]

hey @telamon is this still a WIP? We're really interested in using this kind of feature in peerfs, particularly with regards to permissions-based replication, I can forsee it being really useful. Is there anything that I can do to help get it over the line or finish what you started?

[telamon]

@KGibb8 Yes, this feature is being worked on as we speak. I've reached the point myself where i need fine grained replication control :)

Currently writing a proposal on an "replication-middleware" API that would enable a powerful and elegant way gain control replication for both core-storages and applications. Will leave a comment with a link to the draft as soon as it's published.

[m4gpi]

Great @telamon, I look forward to reading it. When are you aiming to publish it by? Exciting!

[telamon]

@KGibb8 just did!! :tada: https://github.com/telamon/replic8

[telamon]

This PR is super seeded by #32