search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
21 stars 435 forks source link

[SQL Input] Add support for logs in database #9347

Open Alphayeeeet opened 7 months ago

Alphayeeeet commented 7 months ago

I have checked out the technical preview of the SQL-Input integration. Besides metrics, it should also be possible to query log data from SQL-Tables. We have many applications, which log into their respective databases and we would like to monitor those logs as well from Elastic.

It would be nice, if this integration could cover this usecase.

muthu-mps commented 7 months ago

cc: @ishleenk17 , @lalit-satapathy

lalit-satapathy commented 7 months ago

Hi @Alphayeeeet,

SQL input is a generic input, whose goal is to run SQL query against a given database. Are you suggesting the logs are available in a table and can be queried using SQL?

Alphayeeeet commented 7 months ago

Pretty much. So that instead it is only available for metrics, logs could be queried as well. Still there would be the necessity to avoid duplicated query, so the integration should adjust its filter to only query log entries in database since last query or something like that. This could be achieved based on a timestamp column.

A little more in detail:

We have for example database-tables where applications insert log entries as regular datasets. Each dataset equals one log line. All of them go in the same table. The application is cleaning up the database. Therefore only the latest entries since last cleanup are available, but still the integration should periodically query only the new entries since last query.

I hope it is understandable.

lalit-satapathy commented 7 months ago

Got it, we will need some more details on a sample table and DB name for reference. Adding to backlog.