search

orchestracities / ngsi-timeseries-api

QuantumLeap: a FIWARE Generic Enabler to support the usage of NGSIv2 (and NGSI-LD experimentally) data in time-series databases
https://quantumleap.rtfd.io/
MIT License
38 stars 49 forks source link

Extension of the API with the possibility to query NGSI data based on IdPatterns #648

Closed yclausen closed 2 years ago

yclausen commented 2 years ago

When using Quantumleap API for use cases in the company, we noticed that the API is still very limited in its scope for certain queries. The extension of queries based on IdPatterns identically to what can be implemented in ContextBroker would be a positive improvement.

c0c0n3 commented 2 years ago

Hi @yclausen! Thanks for your suggestion!

the API is still very limited in its scope

Yep. QuantumLeap, just like the various FIWARE context broker implementations, takes a "templating" approach to queries where clients can fill in params for a small set of template queries to get data out of the service. In general, this isn't a very flexible approach, b/c the number of queries those templates can produce is a very small subset of all the possible queries you can think of. At the other end of the spectrum, you could provide clients w/ a query languages they can use to write any possible query---e.g. graph ql, sql, etc. This is a more comp-sci sort of approach to the problem, but maybe less practical to implement correctly in Python. (A language like Haskell would be ideal.)

Our thinking is that the template approach could work reasonably well if your templates can handle most of the queries clients would want to use. Surely, a good question is what's that set of most used queries?

The extension of queries based on IdPatterns identically to what can be implemented in ContextBroker would be a positive improvement.

I agree with you. But as you brought it up, I'd like to ask the community a question. How do you get data out of Quantum Leap? Do you use the query API, query the DB directly or both? This would help us narrow that set of most useful queries I mentioned earlier. If we realise the set is too broad, then maybe we should consider the query language approach instead...

yclausen commented 2 years ago

Thank you very much for the quick and detailed reply. From our side I can say that we are mainly focused on using the API of Quantumleap.

c0c0n3 commented 2 years ago

@yclausen thank you so much for the answer!

NEC-Vishal commented 2 years ago

@c0c0n3 so we need to make changes using python?

c0c0n3 commented 2 years ago

@NEC-Vishal good question. I'm more inclined to keep the current templating approach and just add one more feature to the existing template to cater for ID patterns. It could be done w/ a regex, using an approach similar to what we have for the fiware service path at the moment:

@NEC-Vishal, @chicco785 do you agree?

chicco785 commented 2 years ago

@NEC-Vishal good question. I'm more inclined to keep the current templating approach and just add one more feature to the existing template to cater for ID patterns. It could be done w/ a regex, using an approach similar to what we have for the fiware service path at the moment:

@NEC-Vishal, @chicco785 do you agree?

agree!

NEC-Vishal commented 2 years ago

Hi @chicco785 @c0c0n3 yes agree! I have gone through this issue and https://github.com/orchestracities/ngsi-timeseries-api/blob/master/src/translators/sql_translator.py#L822 , I have concluded below points:

  1. We need to update _get_where_clause() in sql_translator.py file.
  2. We need to update all the files like query_NTNE.py or query_1T1ENA.py files (Pass the id_pattern to function calling and the related function)

I will start working as per my understanding and will raise PR for the same. Please correct me if I misunderstood or missed any point.

c0c0n3 commented 2 years ago

Hi @NEC-Vishal,

I think you're on the right track, that's great, thank you so much for this. We'll also need to update the spec with the new parameter

The server reads the spec on startup and makes the parameters available to the Python code accordingly. So without updating the spec, the client won't be able to pass in the id pattern when making the HTTP call. Or more accurately, the client could add it to the URL, but the server would ignore it.

NEC-Vishal commented 2 years ago

Hi @chicco785 @c0c0n3 I have raise a PR (pr648) regarding this issue, Please check the workflow.

Thanks

c0c0n3 commented 2 years ago

Hi @NEC-Vishal! Thanks alot! I commented on #648

c0c0n3 commented 2 years ago

hi @yclausen, @NEC-Vishal implemented the feature you requested and we've just merged #653 which contains the implementation. If you could please give it a try---use the orchestracities/quantumleap:edge image. Any feedback most welcome!

c0c0n3 commented 2 years ago

oh and again kudos to @NEC-Vishal !