serializing deep objects?

[mreinstein]

I didn't see anything in the documentation about nesting serialization functions, or how to handle deep objects with state. Do you have any suggestions for wasabi best practices for this?

[mreinstein]

this seems to work pretty well:

class Power
    constructor: ->
        @max = 12

    serialize: (desc) ->
        desc.uint 'max', 32  # a 32 bit unsigned integer named max

class Player
    constructor: ->
        @x = 0
        @y = 0
        @owned = false
        @p = new Power()
        Wasabi.addObject @p

    serialize: (desc) ->
        desc.uint 'x', 16  # a 16 bit unsigned integer named x
        desc.uint 'y', 16  # a 16 bit unsigned integer named y

    s2cSetControlObject: ->
        @owned = true

Wasabi.addClass Power
Wasabi.addClass Player

it's a bit "blech" though, because now I need to manually track and remove sub-objects before deletion. I'm wondering if instead, Wasabi.addObject and Wasabi.removeObject could traverse the object, looking for serialize functions. Thoughts?

[bkconrad]

I think I want to add a desc.object() method. Maybe something like (pardon my javascript):

Player.prototype.serialize = function(desc) {
    // basically an anonymous serialize function
    // can be used recursively
    desc.object('power', function(desc) {
        desc.uint('x', 16); // you know the drill ...
    }

    // without a function, recursively serializes the whole object via `any`
    desc.object('otherPower');
};

This way it could just reuse the objects that are already there, and you won't have to hang a serialize method on your POD objects.

(side note: this cofeescript stuff looks amazing. I think you just convinced me to try it)

[mreinstein]

I think I want to add a desc.object() method. Maybe something like

That would be great! btw thanks for releasing this library. It's pretty awesome how small and light it is.

this cofeescript stuff looks amazing. I think you just convinced me to try it

I really like it, but it's one of those things that is still very controversial. Many hard core frontend and nodejs people despise it, but I think the syntax is just really nice to look at and easier on the eyes. : ) It also seems to produce "safer" javascript.

[mreinstein]

it just occurred to me that scoping/visibility makes managing these nested serialization objects even more cumbersome. I like your desc.object() proposal because a nested object will only be registered as 1 object, and scope changes wouldn't require manually tracking the nested parts.

[bkconrad]

Ah, that's another really good point. I've actually just started on a .object() implementation with the interface we discussed, and I think it's going to turn out to be pretty useful.

[mreinstein]

wow, awesome! :) That one is important for me because I'm simulating some fairly complex objects over the network, and right now I'm limited to only simulating some of the top level properties.

[bkconrad]

give it a try :)

[mreinstein]

desc.object('power', function(desc) {

yep, this works great! :)

without a function, recursively serializes the whole object via any

I don't understand what you mean here. When I tried calling desc without a function it doesn't seem to work. Are you saying I need to declare a function in my class called any ? If so, not just re-use serialize instead? Or maybe I just misunderstand.

[bkconrad]

I don't understand what you mean here.

Documentation bug! :)

I mean that

// no function passed to desc.object 
desc.object('unstructuredObj');

will automatically serialize all of this structure recursively:

       this.unstructuredObj = {
            uintfoo: 1,
            sintfoo: -1,
            subobject: {
                uintbar: 2,
                sintbar: -2
            }
        };

(example taken from https://github.com/kaen/wasabi/blob/master/test/wasabi.js#L87-L123)

And that it uses InDescription#any and OutDescription#any to deduce, encode, and decode any type of value that Wasabi supports. So you can pay an extra few bits per property in exchange for rapid prototyping.

[mreinstein]

I'm still confused, how is it able to figure out how to deal with desc.object('unstructuredObj'); ? Is it just automatically serializing all sub properties? If so, this might be "too smart". My expectation was given the following structure:

class Power
    constructor: ->
        @max = 12
        @available = 4
        @points = 6

    serialize: (desc) ->
        desc.uint 'max', 32  # a 32 bit unsigned integer named max

class Player
    constructor: ->
        @x = 0
        @y = 0
        @owned = false
        @p = new Power()

    serialize: (desc) ->
        desc.uint 'x', 16  # a 16 bit unsigned integer named x
        desc.uint 'y', 16  # a 16 bit unsigned integer named y
        desc.object 'p'

adding a new Player object would traverse the serialize function and build a bytestream for sending/receiving.

[bkconrad]

Ok, I understand you now.

That's a good idea. Maybe .object can look for a .wsbSerialNumber and just pack a reference (serial number) to the object, then it could be retrieved from the registry on the remote end after normal ghosting/updates. That would save bandwidth if multiple objects refer to the same subobject, too.

I envisioned .object being able to handle POD objects without explicitly declaring their structure, so this is actually a third use case, that would expand .object to have these three uses:

1. Serialize an object using a user-supplied function
2. Serialize an object automatically by traversal
3. Cheaply encode a reference to an object managed by wasabi

Is that still too smart? Maybe 3 should be split out into a separate .reference method?

[mreinstein]

What's POD stand for?

Is that still too smart? Maybe 3 should be split out into a separate .reference method?

Good question. I'm not sure of the answer though. :)

[bkconrad]

I just realized you mean that Power should not be added to Wasabi directly, but should be encoded using the serialize on it. So maybe that's 4 use cases ...

What's POD stand for?

Plain Old Data, just an anonymous Object to use as a data structure

[mreinstein]

I envisioned .object being able to handle POD objects without explicitly declaring their structure

That use case seems a little too magical to me. :) It seems like the general theme of wasabi is that it lets you specify what parts of your object you want to serialize, and how.

This is just my opinion, but the elegant simplicity and small size of your library is very appealing. I'd gladly throw out automagical object serialization if it means keeping the project smaller.

[bkconrad]

The magical recursive serialization really did make a mess of several parts of the code, I think you're right that it should be thrown out.

I also thought of a small problem with the Power example above. If the class isn't added to Wasabi, when it gets pulled out the remote Wasabi instance will only know how to unpack the data if the ghost already has a Power in its p property with a serialize to look for. This is ok in your example, because Players get a Power when they're constructed, but if you changed p on one side to some other type of object (with a different serialize method), or p wasn't set yet, the Wasabi instances will be serializing the data differently, and the universe will implode (the remote end will choke on the Bitstream since it fails to decode it). So Wasabi looking for a serialize method on subobjects is somewhat fragile.

To me, this indicates that the most reliable way to encode subobjects with their own serialize methods is to require that the subobject is added to Wasabi, allowing it to be encoded as a simple reference to an object.

That being said, I think your specific use case could be handled by:

class Player
    # ... snip ...
    serialize: (desc) ->
        desc.object 'p', @p.serialize

Because @p.serialize will always be the correct serialize function here.

So I think we can reduce the functionality of .object down to:

1. Serialize structures explicitly using a function given as an argument
2. Cheaply encode a reference to an object managed by Wasabi

This reliably handles PODs, managed objects, your Power example, supports (explicit) recursion, and doesn't have any surprising automagic, I believe.

[mreinstein]

I also thought of a small problem with the Power example above. If the class isn't added to Wasabi

In my code, I do register Power and Player with Wasabi. I just didn't include it in the example code pasted above. I definitely expect that for any subclass to be serializable it must be addeded to Wasabi first.

[bkconrad]

Oh, my mistake. I'm not sure why I assumed that... I'm going to add a method specifically for objects added to wasabi, named something like netobject, reference, or ghost (thoughts on the name?), and keep object for serializing POD objects with a serialize function.

Anyway, I've flip-flopped again about the automatic recursive serialization. Wasabi already does automatic object traversal in a sense. rpc*Args functions are optional because Wasabi traverses the arguments collection automatically in rpc.js by supplying a special default serialize method which calls desc.any on any values it encounters.

In order for this to keep working, object must either:

1. automatically serialize POD objects if it receives no serialize
2. not support POD objects in RPCs without an Args method

2 leaves a very bad taste in my mouth, because I want Wasabi to have an API layer that is simple without caveat, so default RPC serialization should just work. The basic idea is that Wasabi should operate without serialize methods, and then users can write them as the data structures stabilize. I actually planned to support automatic serialization of managed objects, too (i.e. not requiring a serialize method for classes, either).

So, I think I want a production-grade layer for provided serialize functions, with an "automagic" layer built on top of it by providing a magic default serialize function exactly like in rpc.js. I think the same magic default can be used anywhere a serialize function is expected, meaning that the added code size could be reduced to a single function.

I know this is kind of a radical idea, and I'm as averse to "automagic" as the next person. But if it's applied consistently across the whole library, and has near-zero code complexity cost over the production layer, then I think it's ultimately a good thing from a user's standpoint.

I'm still not set on it yet, so I'd really appreciate your feedback.

[mreinstein]

I can't speak to everyone's motivations, I explored Wasabi after dealing with the frustration of Unity and uLink's networking libraries. They provide very automatic handling of syncing objects and sending RPCs. That comes at the price of flexibility. Wasabi almost instantly enabled me to make my objects network aware, and still maintain a large degree of control over how and when that object syncing occurs.

It seems like there is a struggle between doing more of this automatic handling, and more hand-on control. I don't think there's a right or wrong answer here; it's about defining the philosophy and tone of your particular library.