quickwit-oss / quickwit

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
https://quickwit.io
Other
8.01k stars 327 forks source link

[Feature] Add SQL with Arrow Flight SQL #4052

Open YuriyGavrilov opened 11 months ago

YuriyGavrilov commented 11 months ago

Is your feature request related to a problem? Please describe. Fast sql unified data access.

Describe the solution you'd like Arrow Flight SQL which gives possiblilities to retrieve data faster and also make a one unified standard driver to access.

Describe alternatives you've considered Actually there is no such.

Additional context There is new modern SQL way to retrieve data in a faster way through Arrow Flight SQL driver. There will be no need (it seams to be no) to have python or other kind of clients to access.

https://arrow.apache.org/docs/java/flight_sql_jdbc_driver.html# https://arrow.apache.org/blog/2022/02/16/introducing-arrow-flight-sql/ https://www.postgresql.org/about/news/apache-arrow-flight-sql-adapter-for-postgresql-010-2716/

Some explanations and research: https://www.vldb.org/pvldb/vol10/p1022-muehleisen.pdf

Some server examples: https://github.com/voltrondata/flight-sql-server-example https://www.kamu.dev/blog/2023-09-datafusion-flightsql/ https://github.com/kamu-data/kamu-cli

YuriyGavrilov commented 4 months ago

Still wished to have SQL or Arrow Flight, also good to have https://trino.io/docs/current/connector/elasticsearch.html Trino Elastic compatible API support