orchestracities / ngsi-timeseries-api

QuantumLeap: a FIWARE Generic Enabler to support the usage of NGSIv2 (and NGSI-LD experimentally) data in time-series databases
https://quantumleap.rtfd.io/
MIT License
38 stars 49 forks source link

Document that when using crateDB backend 3D coordinates are not supported #287

Closed glagnar closed 4 years ago

glagnar commented 4 years ago

Describe the bug A valid GeoJson crashes the system. Validation failed for location: Cannot cast {"coordinates"=[53.822510377, 2.485033098, 68], "type"='Point'} to type geo_shape]

Check chapter 9 of the RFC, it is valid to have altitude as a third parameter.

{"type": "Point", "coordinates": [lon, lat, alt]}

To Reproduce Steps to reproduce the behavior:

  1. Send this payload to '...'
  2. See error

Expected behavior A clear and concise description of what you expected to happen.

Environment (please complete the following information):

Additional context Add any other context about the problem here.

chicco785 commented 4 years ago

Hi,

  1. crate does not support 3D coordinates, so in the best case, the processor should strip the alt.
  2. I suspect that some of the functions we use, such as the one to compute centroids, maybe become quite complex computationally.

In short, the only fix in the short term is stripping alt, if that is not going to be good for you, we welcome a pull request to solve the issue otherwise (maybe in timescale 3d support is available).

glagnar commented 4 years ago

Thank you for the speedy reply, and I understand the problem. Here are my thoughts. a) I honestly think it is a bad idea to truncate data without notifying the user, they may loose stuff. b) The user 'me' will find out very quickly if their data is not there, and may find out too late if you save part of the data. c) Improve the error message, to indicate you know you do not support the RFT d) Write somewhere in the docs that you do not support full NGSI ?

Thanks, I will remove the altitude in my payload to get data in.

chicco785 commented 4 years ago

The error message reported is actually from cratedb, the internal processor is perfectly parsing geojson: https://github.com/crate/crate/issues/8053

We will see how to improve error, by clarifying that the error is originated by the backend.

FYI, I checked and postgis supports 3D coordinates, which means also timescale does. Hopefully (haven't tested) this means that if you used timescale as backend, it should work. The draw back would be that we don't have (yet) query api for timescale.

At the time being, I think we will document that the crate backend does not support 3D coordinates.

daminichopra commented 4 years ago

Hi @chicco785 , I would like to work on this issue. Please assign me if no one is working on this issue. Thank you :)

c0c0n3 commented 4 years ago

Hi @daminichopra, just assigned this to you. Thanks alot!

If you could please test what happens with the Timescale backend too. In principle the above example should work with Timescale, but we have to make sure. If it doesn't, then if you could please document that at the moment QL doesn't support 3D coords, regardless of which backend is used.

daminichopra commented 4 years ago

Hi @c0c0n3 , Extension PostGIS can be installed alongside with TimescaleDB. PostGIS supports 3D points, which are created by ST_MakePoint. In short TimescaleDB does support 3D coordinates whereas CrateDB does not support 3D coordinates which i have described in PR. Please check the PR and if it looks good then Merge and close this PR. Thank you :)

c0c0n3 commented 4 years ago

Hi @daminichopra thanks so much for this! @Glagnar so it turns out Crate doesn't support 3D coords and now there's a mention of it in the manual---see #340 by @daminichopra. I reckon we can close this issue then. If you can't do without 3D, you may want to consider using QL with the Timescale backend + PostGIS extension, as @daminichopra points out. Our manual documents how to set up a QL backend made up by Postgres + Timescale + PostGIS. Keep in mind though that at the moment while you can safely use the QL API to insert data into Timescale, the query API still has some rough edges we'll smooth out for the upcoming 0.8 release---see #324.