Can't send number larger than int32

[natcl]

Sending a number larger than 4294967297 seems to fail. From what I understand from the spec numbers larger than 32 bits should be encoded as 2 int32 in big endian format (type h)

[natcl]

More testing:

If I try to send 35314294967297 It gets interpreted as a float like this:

{ "type": "f", "value": 35314294849536 }

[natcl]

To reproduce:

var osc = require('osc');

var packet = {address: '/test', args: 35314294967297};

var result = osc.readPacket(osc.writePacket(packet), {"metadata": true, "unpackSingleArgs": true});
console.log(result); // Prints { address: '/test', args: { type: 'f', value: 35314294849536 } }

[colinbdclark]

Hi @natcl,

Thanks for the note and the example code. In JavaScript, all Numbers are IEEE Floats. There is no native 64-bit integer type. If you want to send large numbers, you'll have to wrap them using the Long library. Here's an example using explicit argument types, but osc.js' type inference will work with Long instances:

var packet = {
    address: "/test",
    args: [
        {
            type: "h",
            value: new Long(0xFFFFFFFF, 0x7FFFFFFF)
        }
    ]
};

Similarly, when you receive "h" typed arguments, they will be Long instances. Check the documentation for the Long library if you want to learn more about how to use it.

[natcl]

Thanks, so it' s normal that any int is sent as float ?

[colinbdclark]

I'm not sure what your question means, but I'll try to clarify. In JavaScript, there are no integers. When you write the number 35314294849536 in JavaScript, it is represented as a float, not an integer. If you don't provide any type metadata about your arguments when you're sending messages, it's not possible for osc.js to reliably distinguish between whether or not an argument should be sent as "i" or "f". If you want to send integers or other non-native JavaScript data types, you must be explicit about the argument types in your payload (as in the example above) in order to ensure that they get correctly encoded in your OSC messages.

If you're writing code that needs to robustly interoperate with any OSC data type, I recommend that you always include argument type metadata when you send messages, and always include the metadata: true flag as well.

I hope this helps!

[natcl]

Yes, thanks for the clarification, so if I want to send a number larger than 4 294 967 296 I'll need to encode it as a Long. Thanks !

[colinbdclark]

Yup, exactly. Glad I could help.

In terms of the issue of type inference, I filed #48 a while ago regarding a potentially compatibility-breaking fix in a future version of osc.js that will turn off type inference by default. I hate the thought of breaking compatibility with existing users, but my impression is that it was a mistake to ship type inference as the default, since it really does get confusing and error prone for users in cases like yours.

[natcl]

And last thing, I'm curious, when sending this: {"type": "i", "value": 4294967295} I get a return value of {"type": "i", "value": -1}

I also tried by sending {"type": "i", "value": 4294967295, "unsigned": true} with the same result. is this normal ?

[colinbdclark]

In the Open Sound Control spec, i-typed values are always 32-bit signed integers. The maximum representable value for a 32-bit two's complement integer is 2147483648. There is no unsigned integer type in OSC, so you'll need to use the h type for large-valued arguments.

[natcl]

Thanks a lot !

[njh]

Hi,

I am looking at a Pull Request njh/node-red-contrib-osc#7 from @natcl which adds Long support to the Node-RED OSC module.

It does this using something like this:

if (typeof value === 'number' && value > LONG_SIZE) {
    packet = {address: path, args: {"type": "h", "value": Long.fromNumber(value)}};
} else {
    packet = {address: path, args: value};
}

However my issue is with separation of concerns - this code has nothing to do with Node-RED but is related to converting from a valid number in JavaScript into correctly typed OSC message. Hence I am wondering why there isn't a rule like this in osc.js?

I can understand that it is not possible to distinguish between integers and floats, but surely then all JavaScript numbers should be output as "d", rather than trying to squeeze it into an "i" resulting in the wrong number being output?

When parsing I can understand converting a OSC type that is bigger than JavaScript can represent natively to a special object type - such a a Long for a 64-bit integer. But it doesn't make a lot of sense to me doing that check on a native integer?

nick.

[colinbdclark]

Hi @njh,

I'm a big fan of Node-RED, so it's great to hear from you!

I think there's still confusion about how osc.js should be used here. In OSC, all message arguments must include explicit type information, no matter what. osc.js, for better or worse, currently supports two different modes for handling this: 1) explicitly typed messages provided by the client, and 2) inferred types, where osc.js makes its best "quacks like a duck" guess at the appropriate argument type to use.

In the case of the former, which I recommended to @natcl as the way to go, all message arguments should include a type field, not just those for int64s. In other words, you should always know and explicitly specify the type of your message arguments in Node-RED. Presumably this is ultimately provided by a user of Node-RED, who is aware of the contract established between them and the target they're sending OSC messages to. As far as I can tell, only they can reliably know when it's ok to coerce a Number to another (foreign to JavaScript) type. The default, otherwise, should always be "d". So, in other words, I agree with your separation of concerns point about this particular logic having "nothing to do with Node-RED," but neither is it something that osc.js can reliably take care of for you. It's application logic!

In the metadata: false mode, osc.js does its best to infer the type of the message arguments provided to writePacket(). This, as I mentioned to @natcl, is convenient for some simple clients of osc.js, but ultimately maybe not robust enough for general-purpose clients such as (I think) yours.

There is never a case where osc.js will attempt to squeeze a JavaScript Number into an OSC "i"-typed argument. Here is the implementation of the inference code:

https://github.com/colinbdclark/osc.js/blob/master/src/osc.js#L1011-L1012

You can see that I do, perhaps erroneously, squeeze JavaScript Numbers into OSC f-typed arguments, which will cause a loss of precision. This is due to the original OSC specification mentioning that the "d" type (i.e. an IEEE 754 float) is "additional" and "non-standard." I felt that falling back to the most interoperable type was wise, though I'm happy to discuss it if others think differently.

In my (admittedly limited) experience with OSC APIs, it's unusual for a message to change its argument types on the fly. As far as I can tell, the current implementation in the pull request you linked to runs the risk of causing subtle issues: if, when sending the same message repeatedly, a user accidentally manages to produce a number that is large enough to trigger the first part of the if/else statement, their messages will suddenly change argument types on the fly, for no predictable reason. But perhaps I'm thinking of OSC APIs too much as statically-typed. Do you know if, in practice in applications such as Lemur or Max/MSP or TUIO, most implementations can roll with an argument suddenly changing type from "f" to "h" and back again? From what I've seen, pretty much every OSC endpoint specifies a single data type for each argument of a message sent to a particular address. I don't feel comfortable, unless I'm missing something really big here, with adding more behaviour to osc.js that will make it harder for users to predict and reason about the contents of the messages they send. Anything that causes a message to change its structure on the fly seems likely to cause hard-to-debug errors with any OSC client that expect static types.

In summary, I think that @natcl's code should always set the metadata flag to true, and it should always specify a type appropriately. At risk of pointing out the now-blazingly-obvious again, there is no native integer type in JavaScript. There's just IEE 754 floats. Presumably the default behaviour should be for Node-RED to set the type of all Number arguments to "d". If your system has some way for users themselves to provide explicit rules regarding how a JavaScript Number should be coerced into another format when sending the message, then this type information should be externalized there. So, for example, some kind of map where users can describe the types of each of the arguments that their Node-RED nodes are sending.

Thoughts?

[natcl]

On my end I don't mind sending an explicit type by detecting what type of data comes in. The current code also detects if the user sends an object with a type defined so the user can always explicitly decide to send a specific type. @njh thoughts ?

[colinbdclark]

Just in case it wasn't clear in my previous post, I want to clarify that I'm not 100% certain of my arguments here, so I'm very much open to hear other opinions as well. Perhaps there are issues I'm missing, or details about how Node-RED works. And thanks, @natcl, for working on this—it's great to hear that Node-RED is using osc.js and that you're making improvements to it.