search

TuGraph-family / tugraph-db

TuGraph is a high performance graph database.
https://tugraph.org
Apache License 2.0
1.43k stars 190 forks source link

Increase Kafka streams support #56

Open jqlin2019 opened 2 years ago

jqlin2019 commented 2 years ago

I have a future request for getting changed data capture from kafka streams when having to query real-time data being from other databases in TuGraph.

I consider adding third-party libraries for TuGraph integration like librdkafka and increase kafka client for polling and executing cypher query.Would you hava plans to support Kafka Connector in future?

ljcui commented 2 years ago

I have a future request for getting changed data capture from kafka streams when having to query real-time data being from other databases in TuGraph.

I consider adding third-party libraries for TuGraph integration like librdkafka and increase kafka client for polling and executing cypher query.Would you hava plans to support Kafka Connector in future?

see this: https://github.com/TuGraph-db/DataX/tree/master/kafkareader, DataX kafka reader

consider using DataX to read kafka and write to tugraph.

jqlin2019 commented 2 years ago

I have a future request for getting changed data capture from kafka streams when having to query real-time data being from other databases in TuGraph. I consider adding third-party libraries for TuGraph integration like librdkafka and increase kafka client for polling and executing cypher query.Would you hava plans to support Kafka Connector in future?

see this: https://github.com/TuGraph-db/DataX/tree/master/kafkareader, DataX kafka reader

consider using DataX to read kafka and write to tugraph.

Thank you for your advice.

However, there will be obvious lags when using DataX to synchronize high-frequency messages because Datax is suitable for batch processing tasks rather than stream processing task.

It makes no sense to turn real-time data into offline data.