Searching and writting data with an SQL query engine?

y-scope / clp

Compressed Log Processor (CLP) is a free log management tool capable of compressing logs and searching the compressed logs without decompression.

https://yscope.com

Apache License 2.0

813 stars 68 forks source link

Searching and writting data with an SQL query engine? #210

Open twoMatches-admin opened 9 months ago

twoMatches-admin commented 9 months ago

Request

I'm looking at CLP and would like to know if you have tried writing an SQL query engine or a query engine for any other declarative language that would make it easier to search over CLP.

Possible implementation

Implementing SQL over CLP using https://github.com/dolthub/go-mysql-server

kirkrodrigues commented 9 months ago

Hey @twoMatches-admin, that sounds interesting. Are you looking specifically for a way to use SQL syntax when querying CLP or a MySQL-compatible interface (such that you can use MySQL client libraries) to query CLP?

If it's the former, we have been looking into providing support for querying using the Kibana Query Language (KQL). Would that be something you're interested in or do you want SQL specifically?
If it's the latter, we hadn't considered that but the library you mentioned does sound promising.
Another area we have been dabbling with off and on is writing a connector for a system like Presto so that Presto users could query CLP.

Let me know which of those sounds most useful to you. On another note, do you have any example queries top of mind that you'd like to run atop CLP?

twoMatches-admin commented 7 months ago

A way to use SQL syntax and already available tools. Specifically SQL to use with in house analytics service that don’t support KQL.

gerilya commented 6 months ago

Hey @twoMatches-admin, that sounds interesting. Are you looking specifically for a way to use SQL syntax when querying CLP or a MySQL-compatible interface (such that you can use MySQL client libraries) to query CLP?

If it's the former, we have been looking into providing support for querying using the Kibana Query Language (KQL). Would that be something you're interested in or do you want SQL specifically?

I think being able to use CLP logs via Kibana is a killing feature if that's what you mean. Elastic is a de-facto standard for logs analysis, so that would help with CLP adoption immensely.

kirkrodrigues commented 6 months ago

Hey @gerilya, are you looking for support for querying CLP data using Kibana Query Language queries or support for using the Kibana UI with CLP?

If it's the former, clp-s has support for KQL. You can try it out using the clp-core container.

If it's the latter, we do have a UI for CLP which you can try by using the CLP package; You'll need to build it from source but we should be making a release in the next week or so. If we were to pursue an integration with Kibana, what features would you say are most important for your use cases?