Closed agaldemas closed 5 years ago
@agaldemas, I think this is only the case for NGSI-LD (@jmcanterafonseca ?), in fact this attribute type is present only in GTFS based models (the only ones already fully encoded into NGSI-LD).
NGSI-LD is not yet supported by QuantumLeap (also in consideration of the fact that there is no a stable release of Context Broker implementation supporting NGSI-LD). Official supported types of NGSIv2 are described in the following spec: http://fiware.github.io/specifications/ngsiv2/latest/
(basically they map to basic schema.org DataType
- https://schema.org/DataType - plus StructuredValue
).
To facilitate the transition, despite it will take sometime before moving to NGSI-LD support, we could think to implement as "StructuredValue" any other "attribute" that is not using a basic datatype. This is anyhow to be carefully evaluated in its implications, as a matter of fact the assumption may be wrong and may lead to undesired behaviours by QuantumLeap.
see also the "ATTRIBUTES DATATYPE TRANSLATION" section in https://quantumleap.readthedocs.io/en/latest/user/#data-insertion
@agaldemas any feedback?
Hello @chicco785, and happy new year 2019.
Sorry for delay, your proposal to implement as NGSI "StructuredValue" any other "attribute" that is not using a basic datatype can perfectly suit our need for PostalAddress attribute. From what I've seen in the translator code (file src/translators/crate.py), it should be easy to achieve, isn'it ? If you want I can issue a PR and try it ? Thanks in advance for your feedback
@agaldemas of course PR are welcome :)
before you investing time on it, I think it would be good to discuss the approach within this issue with @taliaga @c0c0n3
@agaldemas
Is PostalAddress
the type being output by Orion in the notification or you're directly injecting such entity in QL from somewhere else?. If the former, which Orion docker image are you using?
In schema.org, a PostalAddress is a "subtype" of StructuredValue
, so we could start adding this specific translation (adding in NGSI_TO_CRATE an NGSI_POSTAL_ADDRESS
to object
translation).
That should be a relatively doable approach to get your issue fixed, should you need it asap. Your PR will be more than welcome (that's a commonly used entity) and I'm available to help. If you go for it, remember to start with a test, say here.
Later, we can take care of the "generalization" step so as to automatically support other examples of StructuredValues (like PropertyValue, etc.) I'd like to understand first what Orion is planning to do for those. The implementation of this "generalization" should be done considering also discussions like https://github.com/smartsdk/ngsi-timeseries-api/issues/129 and in general a better treatment of incorrect and unsupported types.
Some departing questions: are your PostalAddress
data changing in time? if so, are you planning specific/aggregating queries on top of it?
@taliaga that would be an "hard coded" solution. we can't add a translator for each datatype that is a subtype of StructuredValue
. so for the time being, we just need a procedure that identifies that a type (whatever is named) in the end is a StructuredValue
and store it as that.
To answer @taliaga:
PostalAddress
: we don't specifically think to aggregating queries, our aim is to maximize the use of standardized data models in Orion and historical database.@chicco785 : I agree with your suggestion to have a fallback to StructuredValue (then to object in Crate), for any complex/structured attribute (any subtype of StructuredValue), while keeping its name for the crate column name, since the real interest is to keep historical entity attributes and their values identical to the Orion entity, it should be sufficient, simpler and sustainable.
@agaldemas thanks for your answers and glad to see you'd go for the longer-term solution :)
OK, then how do we proceed ?
I create a PR, or you schedule the enhancement in your backlog ?
I see where it's possible to add the modification, in the file src/translators/crate.py
, function _insert_entities_of_type(), in the for loop on entities attributes:
under the if attr_t not in NGSI_TO_CRATE:
block, test if attribute is a complex one, and if true cast attribute type to NGSI_STRUCTURED_VALUE such as:
table[col] = NGSI_TO_CRATE[NGSI_STRUCTURED_VALUE]
else:
log error message
table[col] = NGSI_TO_CRATE[NGSI_TEXT]
Just I need your help on how you would test if the attribute is complex ? Thanks in advance for your feedback, hence I should be able to issue a PR.
@agaldemas if you are willing to give a try, we will be happy to see a community PR :) Of course, we will support. @taliaga can provide more hints on the way to proceed. But it looks to me your approach is correct, there may be just something more to be considered in the metadata handling, but as first approach sounds ok.
Nice, thanks, I wait for @taliaga hints feedback
Thanks for the PR, I'll comment there.
closed by #142
we have an entity using an address attribute of type PostalAddress such as: { "id": "truc_001", "type": "chose", "address": { "type": "PostalAddress", "value": { "streetAddress": "18 Avenue Félix Faure", "postalCode": "06000", "addressLocality": "Nice", "addressCountry": "France" }, "metadata": {} }, "description": { "type": "Text", "value": "test description", "metadata": {} }, "humidity": { "type": "Number", "value": 88, "metadata": {} }, "location": { "type": "StructuredValue", "value": { "type": "Point", "coordinates": [ 2.185747, 41.3762726 ] }, "metadata": {} }, "status": { "type": "Text", "value": "OK", "metadata": {} }, "temperature": { "type": "Number", "value": 7.22, "metadata": {} } },
When the entity is pushed to QuantumLeap an error occur:
in the FIWARE data models, the type PostalAddress refer to a JSON schema : https://schema.org/PostalAddress and is used as a common reference type for address attribute. So the type PostalAddress should be accepted by the translator (file src/translators/crate.py) as an NGSI type and should translate to an object in crate.