search

c0c0n3 / kitt4sme.live

On a mission to bring AI to the shop floor: https://kitt4sme.eu/
MIT License
1 stars 28 forks source link

Missing Insight time series #186

Closed c0c0n3 closed 2 years ago

c0c0n3 commented 2 years ago

Describe the bug

Insight puts entities into Orion, but there's no corresponding time series in the Quantum Leap DB.

To Reproduce

Spin up a kitt4sme.live instance and port-forward Orion's port, 1026. Then subscribe QL with Orion to receive any context changes for the csic tenant.

$ curl -v localhost:1026/v2/subscriptions \
        -H 'Content-Type: application/json' \
        -H 'fiware-service: csic' \
        -d @quantumleap.subscription.json

where quantumleap.subscription.json is this file. Now simulate Insight sending entity data to Orion.

$ curl -v localhost:1026/v2/entities \
        -H 'Content-Type: application/json' \
        -H 'fiware-service: csic' \
        -d @insight.json

where insight.json is this file. Since csic tenant data gets routed to the default QL DB backend, the entity in insight.json should also be in Crate DB. But if you go look into Crate the entity isn't there. The QL logs tell why

time=2022-08-23 10:46:40.551 | level=ERROR | corr=de8aec54-22d0-11ed-8559-2ecfca3d9c35 | from=127.0.0.6 | srv=csic | subserv=/ | op=_insert_entity_rows | comp=translators.crate | msg=Failed to insert entities because of below error; translator will still try saving original JSON in "mtcsic"."etinsights".__original_ngsi_entity__ | payload=[{'id': 'urn:ngsi:IG:001', 'type': 'Insights', 'Results': {'type': 'StructuredValue', 'value': {'KPI_name': ['Throughput', 'Scrap', 'Roughness'], 'features_names': [['ae', 'fz', 'Diam'], ['Diam', 'fz', 'ae', 'HB'], ['AcelR', 'Ra']], 'features_values': [[2.86, 0.102, 10.21], [14.46, 0.05, 1.36, 88.9], [1.0335, -2.75]], 'KPI_best': ['163.37', '2.0', '1.03']}, 'metadata': {}}, 'time_index': '2022-08-23T10:46:40.193138'}] | thread=139678403169056  | process=7
Traceback (most recent call last):
  File "/src/ngsi-timeseries-api/src/translators/sql_translator.py", line 371, in _insert_entity_rows
    res = self.cursor.executemany(stmt, batch)
  File "/usr/local/lib/python3.8/site-packages/crate/client/cursor.py", line 67, in executemany
    self.execute(sql, bulk_parameters=seq_of_parameters)
  File "/usr/local/lib/python3.8/site-packages/crate/client/cursor.py", line 53, in execute
    self._result = self.connection.client.sql(sql, parameters,
  File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 396, in sql
    content = self._json_request('POST', self.path, data=data)
  File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 523, in _json_request
    _raise_for_status(response)
  File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 206, in _raise_for_status
    raise ProgrammingError(error.get('message', ''),
crate.client.exceptions.ProgrammingError: ElasticsearchParseException[nested arrays are not supported]

Expected behavior

QL should save the Insight entities it receives from Orion to the DB.

Additional context

N/A

c0c0n3 commented 2 years ago

Notice this is more of a limitation of the Elastic Search Crate backend than a Quantum Leap bug. If you route csic tenant data to the QL Timescale backend, the entity time series gets recorded correctly in Timescale.

c0c0n3 commented 2 years ago

So we fixed the problem by