error : PostalAddress is not a supported NGSI Type !

[agaldemas]

we have an entity using an address attribute of type PostalAddress such as: { "id": "truc_001", "type": "chose", "address": { "type": "PostalAddress", "value": { "streetAddress": "18 Avenue Félix Faure", "postalCode": "06000", "addressLocality": "Nice", "addressCountry": "France" }, "metadata": {} }, "description": { "type": "Text", "value": "test description", "metadata": {} }, "humidity": { "type": "Number", "value": 88, "metadata": {} }, "location": { "type": "StructuredValue", "value": { "type": "Point", "coordinates": [ 2.185747, 41.3762726 ] }, "metadata": {} }, "status": { "type": "Text", "value": "OK", "metadata": {} }, "temperature": { "type": "Number", "value": 7.22, "metadata": {} } },

When the entity is pushed to QuantumLeap an error occur:

quantumleap_1  | WARNING:translators.crate:'PostalAddress' is not a supported NGSI type. Please use any of the following: Array, Boolean, ISO8601, DateTime, Integer, geo:json, geo:point, Number, Text, StructuredValue. Falling back to Text.
quantumleap_1  | [2018-12-27 15:10:59,217] ERROR in app: Exception on /v2/notify [POST]
quantumleap_1  | Traceback (most recent call last):
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
quantumleap_1  |     response = self.full_dispatch_request()
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
quantumleap_1  |     rv = self.handle_user_exception(e)
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
quantumleap_1  |     reraise(exc_type, exc_value, tb)
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
quantumleap_1  |     raise value
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
quantumleap_1  |     rv = self.dispatch_request()
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
quantumleap_1  |     return self.view_functions[rule.endpoint](**req.view_args)
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/connexion/decorators/decorator.py", line 66, in wrapper
quantumleap_1  |     response = function(request)
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/connexion/decorators/validation.py", line 122, in wrapper
quantumleap_1  |     response = function(request)
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/connexion/decorators/validation.py", line 299, in wrapper
quantumleap_1  |     return function(request)
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/connexion/decorators/decorator.py", line 42, in wrapper
quantumleap_1  |     response = function(request)
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/connexion/decorators/parameter.py", line 195, in wrapper
quantumleap_1  |     return function(**kwargs)
quantumleap_1  |   File "/src/ngsi-timeseries-api/src/reporter/reporter.py", line 147, in notify
quantumleap_1  |     trans.insert([payload], fiware_s, fiware_sp)
quantumleap_1  |   File "/src/ngsi-timeseries-api/src/translators/crate.py", line 221, in insert
quantumleap_1  |     fiware_servicepath)
quantumleap_1  |   File "/src/ngsi-timeseries-api/src/translators/crate.py", line 318, in _insert_entities_of_type
quantumleap_1  |     self.cursor.executemany(stmt, entries)
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/crate/client/cursor.py", line 67, in executemany
quantumleap_1  |     self.execute(sql, bulk_parameters=seq_of_parameters)
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/crate/client/cursor.py", line 54, in execute
quantumleap_1  |     bulk_parameters)
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/crate/client/http.py", line 328, in sql
quantumleap_1  |     content = self._json_request('POST', self.path, data=data)
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/crate/client/http.py", line 448, in _json_request
quantumleap_1  |     _raise_for_status(response)
quantumleap_1  |   File "/usr/local/lib/python3.6/site-packages/crate/client/http.py", line 187, in _raise_for_status
quantumleap_1  |     error_trace=error_trace)
quantumleap_1  | crate.client.exceptions.ProgrammingError: SQLActionException[ColumnValidationException: Validation failed for address: Cannot cast {"addressCountry"='France', "addressLocality"='Nice', "postalCode"='06000', "streetAddress"='18 Avenue Félix Faure'} to type string]
quantumleap_1  | 172.18.0.11 - - [27/Dec/2018 15:10:59] "POST /v2/notify HTTP/1.1" 500 -
quantumleap_1  | INFO:werkzeug:172.18.0.11 - - [27/Dec/2018 15:10:59] "POST /v2/notify HTTP/1.1" 500 -

in the FIWARE data models, the type PostalAddress refer to a JSON schema : https://schema.org/PostalAddress and is used as a common reference type for address attribute. So the type PostalAddress should be accepted by the translator (file src/translators/crate.py) as an NGSI type and should translate to an object in crate.

[chicco785]

@agaldemas, I think this is only the case for NGSI-LD (@jmcanterafonseca ?), in fact this attribute type is present only in GTFS based models (the only ones already fully encoded into NGSI-LD).

NGSI-LD is not yet supported by QuantumLeap (also in consideration of the fact that there is no a stable release of Context Broker implementation supporting NGSI-LD). Official supported types of NGSIv2 are described in the following spec: http://fiware.github.io/specifications/ngsiv2/latest/ (basically they map to basic schema.org DataType - https://schema.org/DataType - plus StructuredValue). To facilitate the transition, despite it will take sometime before moving to NGSI-LD support, we could think to implement as "StructuredValue" any other "attribute" that is not using a basic datatype. This is anyhow to be carefully evaluated in its implications, as a matter of fact the assumption may be wrong and may lead to undesired behaviours by QuantumLeap.

[chicco785]

see also the "ATTRIBUTES DATATYPE TRANSLATION" section in https://quantumleap.readthedocs.io/en/latest/user/#data-insertion

[chicco785]

@agaldemas any feedback?

[agaldemas]

Hello @chicco785, and happy new year 2019.

Sorry for delay, your proposal to implement as NGSI "StructuredValue" any other "attribute" that is not using a basic datatype can perfectly suit our need for PostalAddress attribute. From what I've seen in the translator code (file src/translators/crate.py), it should be easy to achieve, isn'it ? If you want I can issue a PR and try it ? Thanks in advance for your feedback

[chicco785]

@agaldemas of course PR are welcome :)

before you investing time on it, I think it would be good to discuss the approach within this issue with @taliaga @c0c0n3

[taliaga]

@agaldemas Is PostalAddress the type being output by Orion in the notification or you're directly injecting such entity in QL from somewhere else?. If the former, which Orion docker image are you using?

In schema.org, a PostalAddress is a "subtype" of StructuredValue, so we could start adding this specific translation (adding in NGSI_TO_CRATE an NGSI_POSTAL_ADDRESS to object translation).

That should be a relatively doable approach to get your issue fixed, should you need it asap. Your PR will be more than welcome (that's a commonly used entity) and I'm available to help. If you go for it, remember to start with a test, say here.

Later, we can take care of the "generalization" step so as to automatically support other examples of StructuredValues (like PropertyValue, etc.) I'd like to understand first what Orion is planning to do for those. The implementation of this "generalization" should be done considering also discussions like https://github.com/smartsdk/ngsi-timeseries-api/issues/129 and in general a better treatment of incorrect and unsupported types.

Some departing questions: are your PostalAddress data changing in time? if so, are you planning specific/aggregating queries on top of it?

[chicco785]

@taliaga that would be an "hard coded" solution. we can't add a translator for each datatype that is a subtype of StructuredValue. so for the time being, we just need a procedure that identifies that a type (whatever is named) in the end is a StructuredValue and store it as that.

[agaldemas]

To answer @taliaga:

• the entities are sent to Orion with address attribute of PostalAddress type, and then pushed through notification to QL.
• Orion image is got from dockerhub, dated on 2018-10-31 12:57:48
• departing question about PostalAddress: we don't specifically think to aggregating queries, our aim is to maximize the use of standardized data models in Orion and historical database.

@chicco785 : I agree with your suggestion to have a fallback to StructuredValue (then to object in Crate), for any complex/structured attribute (any subtype of StructuredValue), while keeping its name for the crate column name, since the real interest is to keep historical entity attributes and their values identical to the Orion entity, it should be sufficient, simpler and sustainable.

[taliaga]

@agaldemas thanks for your answers and glad to see you'd go for the longer-term solution :)

[agaldemas]

OK, then how do we proceed ? I create a PR, or you schedule the enhancement in your backlog ? I see where it's possible to add the modification, in the file src/translators/crate.py, function _insert_entities_of_type(), in the for loop on entities attributes: under the if attr_t not in NGSI_TO_CRATE: block, test if attribute is a complex one, and if true cast attribute type to NGSI_STRUCTURED_VALUE such as: table[col] = NGSI_TO_CRATE[NGSI_STRUCTURED_VALUE]

else:
   log error message 
   table[col] = NGSI_TO_CRATE[NGSI_TEXT]

Just I need your help on how you would test if the attribute is complex ? Thanks in advance for your feedback, hence I should be able to issue a PR.

[chicco785]

@agaldemas if you are willing to give a try, we will be happy to see a community PR :) Of course, we will support. @taliaga can provide more hints on the way to proceed. But it looks to me your approach is correct, there may be just something more to be considered in the metadata handling, but as first approach sounds ok.

[agaldemas]

Nice, thanks, I wait for @taliaga hints feedback

[taliaga]

Thanks for the PR, I'll comment there.

[c0c0n3]

closed by #142