Servers and clients need a way to skip flattening/unflattening if not needed

[maffoo]

For servers that just want to pass data along or store it in a lossless way, it would be nice to be able to avoid incurring the cost of flattening/unflattening data.

[pomalley]

Dare I suggest: LazyList?

[maffoo]

LazyList was a half-hearted attempt to do something like this, but it only skipped unflattening lists. I'm thinking of something more drastic here, more inline with how the delphi and java apis work, where you really just have an opaque Data object that is essentially just raw bytes.

[ejeffrey]

Why not just make flattening and unflattening cheap enough that it doesn't matter? Austin's stuff should help with that.

@pomalley: very funny.

I think the right way to do this is to make it explicit. Make a variant of the @setting decorator that allows requesting some or all of the arguments to explicitly stay flattened, wrapped up in an opaque "LabradData" type. Then the server code can either manually extract the data out of it, or just pass it on to an outbound call.

On Wed, Feb 11, 2015 at 7:10 PM, Matthew Neeley notifications@github.com wrote:

For servers that just want to pass data along or store it in a lossless way, it would be nice to be able to avoid incurring the cost of flattening/unflattening data.

— Reply to this email directly or view it on GitHub https://github.com/martinisgroup/pylabrad/issues/65.

[maffoo]

Even if unflattening is cheap, what are you unflattening to? You need a lossless representation of the labrad data, so why not just operate on the bytes directly. Also, as mentioned in #66, it would be nice to uncouple the python representation so the user can unflatten in different ways.

Agree that explicit is better than implicit, and I was thinking along the same lines of modifying the setting decorator. Similarly, could have some kind of flag for clients who make a request and want the result to not get unflattened.

[ejeffrey]

Well, we want a low overhead way to do the flatten/unflatten paths in any case, as that is the 'normal' code path. If we can make that fast enough that we aren't limited by it, and if we can make sure that unflatten+flatten is lossless, then I don't see the point in providing an alternate code path.

On Wed, Feb 11, 2015 at 7:22 PM, Matthew Neeley notifications@github.com wrote:

Even if unflattening is cheap, what are you unflattening to? You need a lossless representation of the labrad data, so why not just operate on the bytes directly. Also, as mentioned in #66 https://github.com/martinisgroup/pylabrad/issues/66, it would be nice to uncouple the python representation so the user can unflatten in different ways.

Agree that explicit is better than implicit, and I was thinking along the same lines of modifying the setting decorator. Similarly, could have some kind of flag for clients who make a request and want the result to not get unflattened.

— Reply to this email directly or view it on GitHub https://github.com/martinisgroup/pylabrad/issues/65#issuecomment-74011354 .

[DanielSank]

You need a lossless representation of the labrad data, so why not just operate on the bytes directly

While this will solve the immediate problem with the datavault and will also help in the case where a server just wants to forward a request or response elsewhere, I think we may be conflating two ideas here. Independently of whether or not we keep the underlying binary data, I'd think we want to guarantee that flatten(unflatten(binary_data)) == binary_data, i.e. promise Losslessness^TM.

Keeping the binary string available seems like a totally independent consideration (although it does provide Losslessness^TM by construction). It's independent because users aren't going to want binary data, they want python objects, so there has to be a lossless binary->python and python->binary mapping anyway.

[ejeffrey]

If we do this, I propose only supporting it in the asynchronous (server)client. That should cover 90% of the use cases, and we can then build it into the packet data structure:

This is what I propose as a syntax for this:

@setting(10, _raw=True, accepts=['_s', '_v[GHz]'], returns=['_i_v[]]) def do_stuff_proxy(self, c, data): p = remote_server.packet() p.do_stuff(data, _raw=True) result = yield p returnValue(p['do_stuff'])

I haven't looked in pylabrad enough recently to know how ugly this is. It definitely crosses all the code in pylabrad, if I recall correctly the unflattering of record data happens already in the packet dissector.

On Wed, Feb 11, 2015 at 10:25 PM, Daniel Sank notifications@github.com wrote:

You need a lossless representation of the labrad data, so why not just operate on the bytes directly

While this will solve the immediate problem with the datavault and will also help in the case where a server just wants to forward a request or response elsewhere, I think we may be conflating two ideas here. Independently of whether or not we keep the underlying binary data, I'd think we want to guarantee that flatten(unflatten(binary_data)) == binary_data, i.e. promise LosslessnessTM.

Keeping the binary string available seems like a totally independent consideration (although it does provide LosslessnessTM by construction). It's independent because users aren't going to want binary data, they want python objects, so there has to be a lossless binary->python and python->binary mapping anyway.

— Reply to this email directly or view it on GitHub https://github.com/martinisgroup/pylabrad/issues/65#issuecomment-74023762 .

[DanielSank]

Why the underscore in _raw?

[maffoo]

@ejeffrey why only for the asynchronous interfaces and what do you mean by "build it into the packet data structure"?

[ejeffrey]

Well, currently the async and syncronous interfaces are largely separate (they use different ServerWrapper and PacketWrapper classes). So I think we should do the async version first at least -- that is where the majority of the use cases are, and that interface is more featureful.

I just mean that we can require using the explicit packet syntax to request non-flattening of return values, rather than allowing it in the direct call syntax. Actually, it could be requested when you construct the packet object if we don't need per-record control over flattening.

I used _raw to prevent confusion with argument names, but I actually think that isn't necessary. In the setting decorator we can tell by context, or we can just have a different decorator. If we put the flag for client calls into packet() it is explicitly separate from arguments.

Evan

On Thu, Feb 12, 2015 at 8:35 AM, Matthew Neeley notifications@github.com wrote:

@ejeffrey https://github.com/ejeffrey why only for the asynchronous interfaces and what do you mean by "build it into the packet data structure"?

— Reply to this email directly or view it on GitHub https://github.com/martinisgroup/pylabrad/issues/65#issuecomment-74103042 .

[maffoo]

Ah, makes sense. When you said "packet data structure" I thought you were referring to bytes on the wire.