How to create ObjectView from binary?

[SupremeTechnopriest]

Hello again! Thanks for the last fix. So I'm hoping to be able to use structurae for my binary protocol, but I'm having trouble reconstructing the view from binary on the client.

Simple example:

// Server
const transfer = TransferMessage.from({
  property: 'a string'
})

ws.send(transfer.buffer)

// Client
const message = TransferMessage.from(event.data)
console.log(message.toObject())

// Prints an empty structure
{ property: '' }

Maybe I'm not doing this right, or maybe its not possible? Its not clear in the documentation. It would be super useful if this were the API to create an object from a buffer.

One additional thing that I would want (but could implement outside structurae) is the ability to tag the buffer with the type of Protocol.

For instance, The first 8 bytes tell you which object to decode the remaining buffer into. Maybe this can be accomplished by wrapping the message? Some tips and insight would be appreciated. Thanks!

[zandaqo]

Hi @SupremeTechnopriest!

Yes, the documentation is lacking--these binary structures are the latest addition and things are still in flux, sorry about that. As for instantiating views, be it ObjectView, ArrayView, etc. from an existing array buffer, just use the constructor:

const transfer = TransferMessage.from({
  property: 'a string'
})

ws.send(transfer.buffer)

// assuming event.data is an ArrayBuffer created by ObjectView
const message = new TransferMessage(event.data)
console.log(message.toObject())
//=> {  property: 'a string' }

Since ObjectView is an extension of DataView, it can be instantiated from existing buffer with offset and byteLength if needed. Just like DataView it "looks" inside the given ArrayBuffer, without copying or creating a new buffer. For another example, when I receive a request in Node.js through HTTP, I instantiate it using the request buffer itself, but I also specify offset and byteLength since the buffer includes other data besides the object I've sent:

app.post('/', (req, res) => {
const message = new TransferMessage(req.body.buffer, req.body.byteOffset, req.body.byteLength);
console.log(message.toObject())
//=> {  property: 'a string' }
});

One additional thing that I would want (but could implement outside structurae) is the ability to tag the buffer with the type of Protocol. For instance, The first 8 bytes tell you which object to decode the remaining buffer into. Maybe this can be accomplished by wrapping the message?

Personally, I rely on additional meta info when figuring out what ObjectView to use to decode the message, such as URL, event names, headers etc. For example, all requests coming to '/houses' route are decoded with HouseView and those coming through '/pets' use PetView. However, if one needs to encode that info into the message itself, I can suggest the following. It's a bit hackish, but bear with me.

First, let's add 'type' field to all our views:

class HouseView extends ObjectView {}
HouseView.schema = { type: { type: 'uint8' }, ... }

class PetView extends ObjectView {}
PetView.schema = { type: { type: 'uint8' }, ... }

Let's say type 1 would be pets, and type 0 are houses. Now, upon creating the view we have to specify the type:

const pet = PetView.from({...})
pet.set('type', 1)

As a side note, it would be nice for ObjectView to have default values and validation, but I'm not yet sure whether to add it as a separate, extending structure or not to avoid the performance hit.

Now, ObjectView just lays out our data in the same sequence as defined in the schema, e.g. if we have 1 byte 'type' (uint8) followed by 10 byte 'name' string, the underlying ArrayBuffer has 11 bytes with the first byte being 'type'. So, on the receiving end we can use DataView to peak into the message first to figure out what ObjectView to use to decode it:

// read the first byte using DataView
const type = new DataView(event.data).getUint8(0);
const message = type === 1 ? new PetView(event.data) : new HouseView(event.data);

Hope this help!

Thank you for the feedback, by the way. I have been using most structures in this library for years, but these binary structures are new. I'm using them in only one project in production and I'm very much curious to see how others would use them, what issues and how they're trying to solve.

[SupremeTechnopriest]

AMAZING! That helps a bunch. We are testing structurae as a serialization layer for our new multiplayer game engine. We will definitely run it through the ringer for you. We are doing our development on discord. I'd be happy to invite you, otherwise we will just drop issues as things come up. We are about to file another issue regarding iterators. Thanks for taking the time to write this out. Maybe you should copy paste it to a markdown somewhere :)

[zandaqo]

That sounds great! I wouldn't want to intrude, but let me know if you have any questions or issues with the library. I'm also using it as a serialization layer in a SaaS that deals with a lot of data from sensors and IoT where JSON is often not supported by clients or would introduce too much overhead. I was hoping to write a tutorial based on my experience, but I'd like to test it more in production.