search

neo4j / neo4j-spark-connector

Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs
https://neo4j.com/developer/spark/
Apache License 2.0
312 stars 111 forks source link

Does this support spark streaming? #78

Closed swayam18 closed 4 years ago

swayam18 commented 5 years ago

Bolt protocol has support for streaming data, so I am wondering, can we use this library to stream data from neo4j and use spark streaming to process it?

jexp commented 5 years ago

Good question, i try to use lazy iterators in all places so in theory it should not fetch data eagerly.

Do you have an example for a spark streaming call? How would it work with RDDs or dataFrames?

kant111 commented 5 years ago

@jexp Since streaming is a micro-batch in spark it will be great if the connector can provide a way to incrementally construct the graph as Spark process every batch.