search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.47k stars 3.99k forks source link

ELK stack integration? #1890

Closed StevenGallagher closed 7 months ago

StevenGallagher commented 3 years ago

Tell us about the new integration you’d like to have

Which source and which destination? Which frequency? Elasticsearch and Logstash - as both Source and Destinations. Realtime (if that's possible)

Describe the context around this new integration

Which team in your company wants this integration, what for? This helps us understand the use case. Many teams use ELK stack for monitoring application logs and metrics. Would be nice to push this into DWH, or pull out of DWH and build Kibana dashboard's.

Describe the alternative you are considering or using

What are you considering doing if you don’t have this integration through Airbyte? Using Kafka + Confluent Kafka Connect - but that's a lot of platforms to maintain.

┆Issue is synchronized with this Asana task by Unito

ChristopheDuong commented 3 years ago

Thanks for your suggestion!

Realtime (if that's possible)

What is an acceptable latency for replication lag that you would accept? (ie the minimum acceptable delay to consider using Airbyte for this?)

(at the moment, Airbyte is not real-time but still has a batch approach)

Would you still be interested if you could do this replication refresh every 5min or 24hours?

stefanondisponibile commented 3 years ago

hey @ChristopheDuong I would be interested in this too. Real time is nice-to-have, but can be lagged, in my case (few hours).

Mostly, it would be interesting having Elasticsearch as a Destination.

Probably I should open a separate issue for this, but is there any guide or possibility of contributing to add custom destinations?

Thank you!

ChristopheDuong commented 3 years ago

hey @ChristopheDuong I would be interested in this too. Real time is nice-to-have, but can be lagged, in my case (few hours).

Mostly, it would be interesting having Elasticsearch as a Destination.

Probably I should open a separate issue for this, but is there any guide or possibility of contributing to add custom destinations?

Thank you!

Hello, Thanks for your interest!

Unfortunately, contributing custom destinations is not well documented yet.

You could follow/comment on the following ticket: https://github.com/airbytehq/airbyte/issues/2755

stefanondisponibile commented 3 years ago

thank you @ChristopheDuong ! ✨

ChristopheDuong commented 3 years ago

you can also have a look at this PR: https://github.com/airbytehq/airbyte/pull/3021

marcosmarxm commented 7 months ago

ElasticSearch is supported.