de-serialization from an array of bytes?

[samtkaplan]

Hello,

What is the best way to de-serialize a FlatBuffers message from an Array{UInt8,1}? The documentation shows how to de-serialize from a FlatBuffers.Builder.

In my use-case, I want to serialize data on one non-Julia process, and then send that serialized data to a Julia process. When I receive the serialized data in the Julia process, it will be a byte-array of some length. So, I need to de-serialize from an Array{UInt8,1} rather than from a FlatBuffers.Builder.

The only way that I can figure to do this is by constructing a dummy copy of my object per the following:

module Foo

using FlatBuffers

type Bar
n::Int32
end

end

using Foo, Base.Test

# create an instance of Bar
x = Foo.Bar(1)

# serialize
xbuilder = FlatBuffers.build!(x)

# assume that we only have access to the bytes, and not the FlatBuffers.Builder instance
bytes = xbuilder.bytes

# make a second instance of Bar. I don't want to do this.  I would prefer to de-serialize `bytes` directly into y.
y = Foo.Bar(999999)

# serialize y
ybuilder = FlatBuffers.build!(y)

# replace ybuilder.bytes with bytes
ybuilder.bytes = bytes

# de-serialize
y = FlatBuffers.read(ybuilder)

@test y.n == 1

[quinnj]

I think the method you're looking for is FlatBuffers.read{T}(::Type{T}, buffer::Vector{UInt8}, pos::Integer) = FlatBuffers.read(Table(T, buffer, pos)). where pos is the "root position" in the flatbuffers table.

[samtkaplan]

Thanks! How do I figure out what the "root position" is? I tried the following:

y = FlatBuffers.read(Foo.Bar, bytes, 0)
@test y.n == 1 # fails!

[quinnj]

If I remember correctly, it's constant by type (w/ # of fields), so you could do:

x = Foo.Bar(1)
xbuilder = FlatBuffers.build!(x)
root = FlatBuffers.get(xbuilder, xbuilder.head, Int32)

# then
y = FlatBuffers.read(Foo.Bar, bytes, root)

[samtkaplan]

Thanks...but that gives the following error:

LoadError: BoundsError: attempt to access 24-element Array{UInt8,1} at index [-262139:24]

root evaluated to 12 for this example.

[quinnj]

Hmmm.....obviously I'm remembering that wrong. Let me try to figure out how to do this.

[quinnj]

So the example I posted does actually work, but if you're trying to use the Julia-side byte buffer, you have to do

x = Foo.Bar(1)
xbuilder = FlatBuffers.build!(x)
root = FlatBuffers.get(xbuilder, xbuilder.head, Int32)

# then
y = FlatBuffers.read(Foo.Bar, xbuilder.bytes[xbuilder.head+1:end], root)

i.e. the Julia byte buffer has some padding to ensure certain alignments, but if you're looking for just the raw flatbuffer data in the buffer, you do xbuilder.bytes[xbuilder.head+1:end].

[quinnj]

Additionally, the root position is given in the first 4 bytes of a flatbuffer buffer, so a general purpose flatbuffer buffer reader could be

FlatBuffers.read{T}(::Type{T}, bytes) = FlatBuffers.read(T, bytes, read(IOBuffer(bytes), Int32))

I'm going to add that definition to the package.

[samtkaplan]

Thanks!!