Handling object references

[kumavis]

I'm looking to do something like this:

// on one machine
var syncedObjectStore = new SyncedObjectStore(duplexStream) 
var objA = new syncedObjectStore.Object()
var objB = new syncedObjectStore.Object()
var objC = new syncedObjectStore.Object()

objA.set('b',objB)
objB.set('c',objC)

// on another machine
var syncedObjectStore = new SyncedObjectStore(duplexStream)

// later we have a reference to these objects created remotely and
// we can see their references have been kept
atSomePointLater(function(objA,objB,objC){
  objB.get('c') === objC //=> true
})

Notice how object references are maintained. I think this would go a long way to making a distributed system feel like a single machine.

This might require automatically registering normal objects into the SyncedObjectStore

// on one machine
var objD = {greeting: 'hello!'}
objA.set('d',objD)
objB.set('d',objD)

// on another machine
objA.get('d') === objB.get('d')

[kumavis]

It Would Be Easy™ to add this functionality on top of scuttlebutt, but I'm curious if this already exists.

[dominictarr]

No, this doesn't already exist. Although,my replicated data store that predated scuttlebutt - snob (which was a git-esque hash tree of diffs) I had a thing to maintain the structure of references, and detect cycles, etc.

(although, this required that each branch had and __id__ property, although you could just use weakmaps)

scuttlebutt was my 3rd attempt at a data replication system, each simpler than the one before it.

I don't mean to be discouraging, but I think it's unwise to try to pave over distribution and make it feel like a single system. I think the bible has a section on distributed programming which says something like "God, grant me the serenity to accept the things I cannot change, The courage to change the things I can, And wisdom to know the difference"

I think distributed systems is one of these things you cannot change. It's a much better base assumption that you are running across many systems. The way things are going (massively multicore) sooner or later people will be using eventually consistency techniques within a single (multicore) chip!

That is not to say that a scuttlebutt that replicates a tree structure - which is what this boils down to, i think? is not an interesting and worthwhile problem.

What do you want to use it for?

[kumavis]

Ah hah yes, wise words. I did have a feeling in the back of my mind that it would be an over simplification and paving over of the subtleties of the system.

As for why and what for, its really just an experiment. Mad science. I'm new to thinking about things as distributed systems and was inspired by scuttlebutt. I've been playing with voxel.js and multiplayer over webrtc, but started thinking of a system without a dedicated server.

Anyways I'll throw it together and see what its like to work with.

[dominictarr]

Okay, awesome! let me know if you have any questions!

Just on thing though:

//this pattern is better
duplex.pipe(sb.createStream()).pipe(duplex)

//than:
sb.handle(duplex)

Because you know that your thing is using streams by the correct api and not doing anything weird. There is now quite a range of things that use this pattern, from dnode, rpc-stream to scuttlebutt and it's subclasses.

[kumavis]

+1 to that. Also frees your module consumer to pipe things in ways you didn't expect (like a throughput benchmarking transform).

[kumavis]

Heh, here it is: https://github.com/kumavis/synced-object-store

Unfortunately it doesn't work out of the box for things like handing a voxel.js game object, because of a reference to window somewhere in the tree.

While I can't imagine using this in a real project, 'twas a fun hack. Curious what your thoughts are.

[dominictarr]

@kumavis looks interesting. can you put a simple example of how to use it in the readme?