Data type conversion for scalar value types from proto3 to JSON compliant datatypes

[S-Eibl]

Feature

Actual implementation

The actual version of the CLI accepts only basic data types for scalar field types:

func fieldToDataSchema(f *proto.Field) wot.DataSchema {
    switch f.Type {
    case "bool":
        return wot.DataSchema{DataType: "boolean"}
    case "float":
        return wot.DataSchema{DataType: "number"}
    case "int32":
        return wot.DataSchema{DataType: "integer"}
    default:
        return wot.DataSchema{DataType: "string"}
    }
}

Goal

The CLI should be able to transform every scalar value type available for proto3 syntax denoted by https://developers.google.com/protocol-buffers/docs/proto3#scalar.

Acceptance criteria

• Each available scalar value type can be parsed
• The fields scalar type is converted to the correct type for JSON according to the table

[iomz]

The fields scalar type is converted to the correct type for JSON according to the table

As I commented in #14, there's no JSON in the table...?

[S-Eibl]

Thank you for the hint. I'm sorry, this was too unprecise. The proto3 types can define variables more precisely than the allowed JSON datatypes (for the TD they are listed for DataSchemes https://www.w3.org/TR/wot-thing-description/#dataschema). All float and double types have to be abstracted to numbers. All integer types have to be transformed into integers. String and bytes are converted to string in JSON. The complete table for our purpose denotes as follows:

[iomz]

All float and double types have to be abstracted to numbers. All integer types have to be transformed into integers. String and bytes are converted to string in JSON.

To be clear, this mapping is then "your suggestion", am I right?

[S-Eibl]

Yes, based on the available data types for the TD and the possible types in proto syntax. I think for "string", "bytes", and "bool" it is clear. "double" and "float" are numbers, as this is the JSON representation for floating points. For all int values up to 32 bit, it is an integer value, as passing with floating-point might lead to uncertainty. The main problem are 64 bit integers. Javascript and JSON cannot represent 64 bit integers as far as I know, so they would have to be represented as strings or numbers. For numbers, there is still the uncertainty problem and for strings, the format would have to be set for the client. For simplicity reasons I assigned them to integers for now, but I'm not confident with this solution. Do you have an idea?

[iomz]

I agree with the mapping, although this is a lossy conversion from ProtoBuf to TD, and irreversible from TD to ProtoBuf (which undermines the original purpose of this tool to automate the creation process to improve the integrity). Is there any way to denote the original type in ProtoBuf in TD's DataSchema?

P.S. I guess last time I read your thesis draft, you wrote "it was taken from gRPC doc", but then that was wrong...

[S-Eibl]

There is a conversion policy at the proto3 docs, but it converts 64-bit numbers to strings. The problem with the Thing Description arises as it should express the type and format of a property. I would suggest bypassing this through xsd types. Then the originator can emit values for any kind, and the recipient can format it according to the "xsd" type. I would therefore change the table from above accordingly: Denoting the original type from proto syntax in TD's DataSchema would require an ontology for the proto3 scalar types, and I haven't found one which supports that. The other possibility would be to write in the proto3 types directly in the DataSchema. This only works if the client understands these types and a conversion schema for proto3 types is added in the nodewot. So the possibilities are:

• using the table from proto3 docs
• using xsd types
• developing an ontology for proto3 scalar types
• expecting the client to be able to operate proto3 syntax and directly pasting the types

I prefer the xsd types; what do you think?

[iomz]

I'm not how the xsd annotations would solve the irreversible translation problem.

The other possibility would be to write in the proto3 types directly in the DataSchema. This only works if the client understands these types and a conversion schema for proto3 types is added in the nodewot.

Why nodewot?

To do this in a proper way, I'd create a vocabulary for ProtoBuf's scalar value types. But since you are severely backlogged from the planned milestone now, I suggest you to keep the mapping as is and move foreward.