iotivity-node should preserve the type of numbers in the payload

[ttzeng]

It seems iotivity-node will send out integer numbers in the property payload if the floating numbers in the committed payload are fixed numbers. For example, if modify the ambient_light.js in the SmartHome demo to send out fixed illuminance value 'lux = 10.0' in the property to observers, the observers will get the illuminance property of type integers instead of floating numbers.

[gabrielschulhof]

If you build and run https://gist.github.com/gabrielschulhof/a1b1fa02708f68b15160dce3ac36a449, you will see that JS turns floating point numbers of integer value into integers. The script below is part of that gist:

var hello = require( "bindings" )( "hello" );

function doOneNumber( theNumber ) {
    console.log( theNumber + ": " + JSON.stringify( hello.getNumberType( theNumber ) ) );
}

doOneNumber( -1.2 );
doOneNumber( -1.0 );
doOneNumber( -1 );
doOneNumber( -0.0 );
doOneNumber( -0 );
doOneNumber( 0 );
doOneNumber( +0 );
doOneNumber( +0.0 );
doOneNumber( 0.0 );
doOneNumber( 1.0 );
doOneNumber( +1.0 );
doOneNumber( +1 );
doOneNumber( 1 );
doOneNumber( 1.2 );
doOneNumber( +1.2 );

Its output:

-1.2: {"IsNumber":true,"IsUint32":false,"IsInt32":false}
-1: {"IsNumber":true,"IsUint32":false,"IsInt32":true}
-1: {"IsNumber":true,"IsUint32":false,"IsInt32":true}
0: {"IsNumber":true,"IsUint32":false,"IsInt32":false}
0: {"IsNumber":true,"IsUint32":false,"IsInt32":false}
0: {"IsNumber":true,"IsUint32":true,"IsInt32":true}
0: {"IsNumber":true,"IsUint32":true,"IsInt32":true}
0: {"IsNumber":true,"IsUint32":true,"IsInt32":true}
0: {"IsNumber":true,"IsUint32":true,"IsInt32":true}
1: {"IsNumber":true,"IsUint32":true,"IsInt32":true}
1: {"IsNumber":true,"IsUint32":true,"IsInt32":true}
1: {"IsNumber":true,"IsUint32":true,"IsInt32":true}
1: {"IsNumber":true,"IsUint32":true,"IsInt32":true}
1.2: {"IsNumber":true,"IsUint32":false,"IsInt32":false}
1.2: {"IsNumber":true,"IsUint32":false,"IsInt32":false}

As you can see, the intention that a number be floating point is not preserved even as it is passed to a JS function, much less as it is passed to the native side.

We need to decide whether endpoints are expected to accommodate a variety of mutually compatible data types for a given property or, conversely, whether the protocol is to be fixed and immutable, meaning that a given property must have exactly one agreed-upon data type or else be considered invalid.

This requires some discussion, research, and elucidation.

@poussa, @zolkis, @siovene, @nagineni

[zolkis]

In the illustrated use case (10.0 becoming 10) I don't see the user of that value (observers) really affected. If the data model of that resource requires a floating point, and the observers get 10 as integer, it is a valid representation of the property.

In this version of the API, and IIUR in OCF in general, handling (interpreting) the data model belongs to the applications, i.e. the app will get 10 from the OCF implementation, then checks the data model and displays 10.0 instead of 10. But even if it displays 10, it is not an error. Important is to accept e.g. 10.3 as a new value, and that is not a problem.

The alternative is to provide API primitives that convey type information, like resource.setValue("temperature", "23.0", "float", "C");, providing property name, value, type, and semantic annotation(s) for resource representation.

I'd like to keep it the simple way of apps owning the data model, and using simple property setters in scripts: resource.properties.temperature = 23.0;

If it proves to be needed, we can add the semantic setters in a later version. Even then, the current model will be still a legal way to specify values.

[ttzeng]

The OCF spec also specify the type of property that the observers should expect. Take illuminance resource for example, the spec: " illuminance is a float and represents the sensed luminous flux per unit area in lux", observers invokes iotivity C++ API OC::OCRepresentation::getValue() or Java API OCRepresentation.getValue() will follow the spec and cast the value to the desired type. The C++/Java observers will program abort or get incorrect data if iotivity-node/JS converts floating point to integer automatically.

[zolkis]

The C++/Java observers will program abort

Why would the program abort?

or get incorrect data if iotivity-node/JS converts floating point to integer automatically.

I don't think so. It's still a valid value for the representation on the reading side. When 10.0 is expected, 10 is a fine value.

[ttzeng]

My OCF client uses Java API "double illuminance = (double) OCRepresentation.getValue(key)" to get the updates from the ambient_light.js. Since the payload contains the property attributes, so it throws OcException.

[zolkis]

That sounds like a bug on the implementation side. It should be able to convert an integer to a double.

[ttzeng]

No, java.lang.Double can't be cast to java.lang.Integer. I got exactly the same error as this post. BTW, my OCF client is an Android app leveraging IoTivity Android base library.

[zolkis]

It seemed to me that you want to cast from integer to double in the code, not the other way around.

[ttzeng]

You are right, I quoted the wrong post. In fact the representation convoys Objects with key 'illuminance', according to the spec, I thought they are doubles, but some of them are integers and throws exception.

[zolkis]

Even then, implementations should handle this corner case, instead of making requirements on how JavaScript should handle types... :)

[ttzeng]

Does it mean iotivity-node can't preserve the given property types as it's JS limitation? There are C++ and Java APIs OcRepresentaton.setValue(...) overloaded for composing representation properties of each primitive types to pass the properties with desired types, so the observers receive these property objects have deterministic way to interpret the data.

Yes, I have added logic in my Android client to check whether the property is an instance of integer or double and interprets the object separately, but I am hoping inotivity-node could preserve the object attributes as other languages do make the interoperability better. Thanks. :)

[zolkis]

Does it mean iotivity-node can't preserve the given property types as it's JS limitation?

That is about right, this is a limitation of JavaScript: you should not infer strict type information from values received from JavaScript, but you need to consult metadata (or documentation) for knowing for sure what type is expected.

It is a separate discussion whether to include the metadata with the representation, or whether to have programmatic support for downloading metadata when in doubt. Let's see if/when OCF gets there, but it will be there in WoT.

Anyway, in this version of the API we are not doing it, but wait for where the standards get with that. We can create an issue to track this and we can label it with "future work" or "V2".

[gvancuts]

Hi @zolkis , if I understand the issue correctly, this makes any OCF server implemented in JavaScript (iotivity-node) impossible to certify against the OIC specifications, is this correct? If true, this seems like a serious limitation to me.

[zolkis]

this makes any OCF server implemented in JavaScript (iotivity-node) impossible to certify against the OIC specifications

If true, it would indeed be a serious limitation, but I would question the validity of the test in question.

Could you share a link to the failing OIC certification test case(s)?

I just don't understand why would this fail certifications. If a server is expected to send float 10.0 in CBOR, and it sends unsigned int 10, or int 10, it can be still successfully parsed as float 10.0 on the receiving end when required so by the data model.

I didn't find any clause in CBOR RFC that would say otherwise. Conversely, it says this in 1.1.1:

There is no requirement that all data formats be uniquely
          encoded; that is, it is acceptable that the number "7" might
          be encoded in multiple different ways.

[gvancuts]

I haven't tried to run OIC certification test cases so I don't know if any of these tests would fail. Have you done that exercise as part of your development?

My statement was purely based on my reading of the specs, where they indicate the value should be a float and not an int. What you're saying is that it could be sent as an int on the wire but then should be interpreted as a float, is that correct?

[zolkis]

What you're saying is that it could be sent as an int on the wire but then should be interpreted as a float, is that correct?

Yes.

I have also checked the OCF Core spec, and it relies on RFC 7049 linked above. The formulation is generic:

If a property is defined as a JSON number an implementation shall accept integers, single- and double-precision floating point.

I am pretty sure the expected behavior on the receiving client side is to convert integer to float when the data model requires float and the server sends integer.

[gabrielschulhof]

Indeed, you can clone the repo containing the data models and see that data types are either integer, so if a float appears you can legitimately throw an error, or number, which is governed by the statement quoted by @zolkis above.

$ cat .git/config | grep url
    url = git@github.com:OpenInterConnect/IoTDataModels
$ git grep '"type":' | \
> sed 's/[{},[:space:]]//g' | \
> awk -F ':' '{for(Nix = 1; Nix <= NF; Nix++)
> if ( $Nix == "\"type\"" ) {Nix++; print $Nix; break};}' | \
> sort -u

"array"
"boolean"
"integer"
["integer""string"]
"number"
"object"
"string"

Since the data models never explicitly expect a float, but rather always numbers, therefore I believe that the data models have Javascript numeric semantics, and this problem is reduced to the case where you're expecting an integer but you're receiving a float, because the application on the other end mistakenly sent a float, and iotivity-node leaves it up to the application to validate the data against the relevant data model.

I believe this issue can be closed, but I leave it up to others to do so.