Would be nice to have a json schema for each data model

LeonanCarvalho commented 7 years ago

The documentation is easily readable to humans. Thinking about machine-readable concept would be nice to have a file containing the JSON data model schema because markdown isn't a good structure to do that, anyway using this format could be easily consumed by developers also.

Despite having a pretty good documentation, references and links, there is no kind of support to developers or ways to automatize and consumers of this standardize the data models as schemes or another way to be a machine-redabe model. Data Models there's no limits, new models and changes on currently ones will be necessary along time. It will evolve aligned to new discoveries, news technologies. So, I think Data Models should be flexible, with version control and scheme standard files on a own repository to be consumed or "learned" by smart applications and developers through a know semantic.

Extracting from the docs, I could build a scheme with all attributes of the Device Data Model. I tried to cover all specs to be consumed by an application. The result was:

{
    "namespace": "Environment",
    "name": "WaterQualityObserved",
    "attributes": {
        "id": {
            "descr": "Unique identifier.",
            "type": "Text",
            "mandatory": true
        },
        "type": {
            "descr": "Entity type. It must be equal to WaterQualityObserved.",
            "type": "Text",
            "allowedvalues": "WaterQualityObserved",
            "mandatory": true
        },
        "location": {
            "descr": "Location where measurements have been taken, represented by a GeoJSON Point.",
            "type": "geo:json",
            "allowedvalues": {
                "normative": "https://tools.ietf.org/html/rfc7946"
            },
            "mandatory-if": [
                {
                    "attribute": "address",
                    "rule": "equals",
                    "value": "undefined"
                }
            ]
        },
        "address": {
            "descr": "Civic address where the Water Quality measurement is taken.",
            "type": "address",
            "allowedvalues": {
                "normative": "https://schema.org/address"
            },
            "mandatory-if": [
                {
                    "attribute": "location",
                    "rule": "equals",
                    "value": "undefined"
                }
            ]
        },
        "temperature": {
            "descr": "Temperature.",
            "type": "Number",
            "metadata": {
                "timestamp": {
                    "type": "DateTime"
                }
            },
            "unit": {
                "symbol": "CÂº",
                "descr": "Celsius Degrees"
            }
        },
        "conductivity": {
            "descr": "Electrical Conductivity.",
            "type": "Number",
            "metadata": {
                "timestamp": {
                    "type": "DateTime"
                }
            },
            "unit": {
                "symbol": "S/m",
                "descr": "Siemens per meter"
            }
        },
        "conductance": {
            "descr": "Specific Conductance.",
            "type": "Number",
            "metadata": {
                "timestamp": {
                    "type": "DateTime"
                }
            },
            "unit": {
                "symbol": "S/m",
                "descr": "Siemens per meter at 25 ÂºC"
            }
        },
        "tss": {
            "descr": "Total suspended solids.",
            "type": "Number",
            "metadata": {
                "timestamp": {
                    "type": "DateTime"
                }
            },
            "unit": {
                "symbol": "mg/L",
                "descr": "milligrams per liter"
            }
        },
        "tds": {
            "descr": "Total dissolved solids.",
            "type": "Number",
            "metadata": {
                "timestamp": {
                    "type": "DateTime"
                }
            },
            "unit": {
                "symbol": "mg/L",
                "descr": "milligrams per liter"
            }
        },
        "turbidity": {
            "descr": "Amount of light scattered by particles in the water column.",
            "type": "Number",
            "metadata": {
                "timestamp": {
                    "type": "DateTime"
                }
            },
            "unit": {
                "symbol": "FTU",
                "descr": "Formazin Turbidity Unit"
            }
        },
        "salinity": {
            "descr": "Amount of salts dissolved in water.",
            "type": "Number",
            "metadata": {
                "timestamp": {
                    "type": "DateTime"
                }
            },
            "unit": {
                "symbol": "ppt",
                "descr": "Parts per thousand"
            }
        },
        "pH": {
            "descr": "acidity or basicity of an aqueous solution.",
            "type": "Number",
            "metadata": {
                "timestamp": {
                    "type": "DateTime"
                }
            },
            "unit": {
                "symbol": "",
                "descr": "Negative of the logarithm to base 10 of the activity of the hydrogen ion."
            }
        },
        "orp": {
            "descr": "Oxidation-Reduction potential.",
            "type": "Number",
            "metadata": {
                "timestamp": {
                    "type": "DateTime"
                }
            },
            "unit": {
                "symbol": "mV",
                "descr": "millivolts"
            }
        },
        "O2": {
            "descr": "Level of free, non-compound oxygen present.",
            "type": "Number",
            "metadata": {
                "timestamp": {
                    "type": "DateTime"
                }
            },
            "unit": {
                "symbol": "mg/L",
                "descr": "milligrams per liter"
            }
        },
        "Chla": {
            "descr": "Concentration of chlorophyll A.",
            "type": "Number",
            "metadata": {
                "timestamp": {
                    "type": "DateTime"
                }
            },
            "unit": {
                "symbol": "mg/L",
                "descr": "milligrams per liter"
            }
        },
        "PE": {
            "descr": "Concentration of pigment phycoerythrin which can be measured to estimate cyanobacteria concentrations specifically.",
            "type": "Number",
            "metadata": {
                "timestamp": {
                    "type": "DateTime"
                }
            },
            "unit": {
                "symbol": "mg/L",
                "descr": "milligrams per liter"
            }
        },
        "NH4": {
            "descr": "Concentration of ammonia.",
            "type": "Number",
            "metadata": {
                "timestamp": {
                    "type": "DateTime"
                }
            },
            "unit": {
                "symbol": "mg/L",
                "descr": "milligrams per liter"
            }
        },
        "Cl": {
            "descr": "Concentration of chlorides.",
            "type": "Number",
            "metadata": {
                "timestamp": {
                    "type": "DateTime"
                }
            },
            "unit": {
                "symbol": "mg/L",
                "descr": "milligrams per liter"
            }
        },
        "NO3": {
            "descr": "Concentration of nitrates.",
            "type": "Number",
            "metadata": {
                "timestamp": {
                    "type": "DateTime"
                }
            },
            "unit": {
                "symbol": "mg/L",
                "descr": "milligrams per liter"
            }
        },
         "dateModified": {
            "descr": "Last update timestamp of this entity.",
            "type": "DateTime"
        },
    }
}

namespace : Represent the universe domain/group/universe from a DataModel.
name : DataModel Name.
attributes : Dataset of a DataModel, each key represents an attribute from DataModel, using a StructuredValue to represent properties:
- type : Represent a Data Type, could be a Schema's Data Type or another DataModel. The Value can be a Text or a List of Text to represent mixed values, (obs. : I used TextList to represent Arrays because I do not find the correct schema datatype to represent a List of values)
- unit : Structured Value composed by symbol and a description
- metadata : Structured Value with same structure from the main structure, but metadata could not have a nested metadata attribute.
- allowedvalues : A Structured Value, List or Text to flexibilize all posibilities , data model reference, normative reference and possible values. Default. "*"
- from: URL or URI to from referenced datamodel or context.
- attribute : Text representing the attribute name from the reference.
- interval: A list of 2 Numeric Values that represent a range of Numeric Values. e.g. [0.0,1.0].
- normative : URL with normative reference
- mandatory : Boolean value to represent if the attribute should aways exists on referenced datamodel (defaulf. false)
- mandatory-if : A list of Structured Values with attribute (name of attribute that rule applies), rule(rule description) and value (Value based on rule that define if is mandatory ). eg.
```
(...)
"mandatory-if":[
{"attribute": "category", "rule": "contains", "value":["tracked"]},
{"attribute": "category", "rule": "no-contains", "value":["specialUsage"]}
]
(...)
```

The other option could also be used a jsonld like https://schema.org/docs/tree.jsonld, that's had a much richer and complex structure.

I believe that it is not possible to speak of a harmonized data model without be closer of the ontology concepts. But this proposal isn't intended as a universal ontology for this harmonized datasets, but to provide ways to developers implement it with more fluidity. I made a fork and I'll work on a structured json to be consumed by our Fiware Powered applications to evaluate this model applied on our concepts.

jmcanterafonseca commented 7 years ago

thanks for this input. I think it would be great to have a JSON, machine readable description of each of the data models. That would eventually allow us to automatically generate the markdown documentation file associated to each data model.

Any contribution would be very welcome. For instance, you can start by contributing the example you have described in this issue.

LeonanCarvalho commented 7 years ago

Thank your for the appreciation.

I'll send a pull request and I'm open to discus this model with you. I'll include Professor @flaviosilva on this conversation, may he could help us to improve and review this schemas. On this pull I'll send the schemas for Device, DeviceModel, WaterQualityObserved, Vehicle, VehicleModel.

jmcanterafonseca commented 7 years ago

thanks Leonan, but please use one pull request per entity type, that would make it easier the review process

best

2016-10-31 14:21 GMT+01:00 José Leonan S. Carvalho <notifications@github.com

:

Thank your for the appreciation.

I'll send a pull request and I'm open to discus this model with you. I'll include Professor @flaviosilva https://github.com/flaviosilva on this conversation, may he could help us to improve and review this schemas. On this pull I'll send the schemas for Device, DeviceModel, WaterQualityObserved, Vehicle, VehicleModel.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Fiware/dataModels/issues/76#issuecomment-257291581, or mute the thread https://github.com/notifications/unsubscribe-auth/AAqZ_lJulfMaebdXK8vHtY1EX_h_onHXks5q5es9gaJpZM4KgE9O .

flaviosilva commented 7 years ago

@LeonanCarvalho and @jmcanterafonseca ,

It really makes sense a DataModel Schema. In some way, this remembers me the Web Services Description Language (WSDL) - https://www.w3.org/TR/wsdl - WSDL is used to describe an Web Service in a machine readable format. We are open to contribute to this initiative!

I would suggest follow guidelines defined under IETF to the schema (https://tools.ietf.org/id/draft-zyp-json-schema-04.html). There are related initiatives regarding JSON Schema (http://json-schema.org/). There is also a tool that can help the JSON Schema creation (http://jsonschema.net/)

LeonanCarvalho commented 7 years ago

That's the point! Thanks!

The guidelines defined under IETF should be enough to clarify some questions, like how to represent normative references of attributes.

I would like to read it deep this requirements to better understand and then adjust the models suggested on #81 , #82 , #83 , #84 and #85 pull requests.

On a second moment it will be great develop a tool to generate the structure and the docs based on this schema.

jmcanterafonseca commented 6 years ago

closed as dup of #151

FIWARE / data-models

Would be nice to have a json schema for each data model #76