search

redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8.13k stars 834 forks source link

Support for neo4j #869

Closed E22T07 closed 1 month ago

E22T07 commented 3 years ago

Hi, it would be great if Benthos had support for Neo4j.

Thanks!!

jem-davies commented 1 year ago

Other graph-based DBs are using Neo4j's Cypher Query Language, for example memgraph, so the implementation of this should add connectivity to a few graph-based DB engines.

I am going to take a look at SQL based processors for inspiration - I think that it will be hard to generalise converting JSON messages into a Graph DB. - Does anyone have any ideas / examples in other software that does something similar?

I think that the processor should be named Cypher so it similar to the other SQL processors (that the query language is used for the name not the DB engine i.e. Neo4j.)

jem-davies commented 1 year ago

from @boorad on the discord server:

I am planning on open-sourcing all of our ArangoDB input/processor/output Benthos components soon. I prefer this graph db to Neo4j... not sure if you have any interest.

jem-davies commented 1 year ago

@boorad I think that it is interesting because there are many graph DBs out there. They have different query languages and I couldn't find anything like an ORM so that it could be general across the different graph DBs. (although I haven't searched extensively)

Looking at benthos's SQL components it makes use of the ORM to generalise between SQL dialects. -- that would be ideal for graph DBs but it might not be possible.

Neo4j's query language has been open sourced and is implemented in more that just neo4j. I think that my intention is to provide a group of Cypher components so it is clear that if the target Graph DB uses Cypher it should work.

We see that this memgraph uses the neo4j-go-driver https://memgraph.com/docs/memgraph-cloud/cloud-connect#golang .

....

It would be interesting to see the components you have so that we could work out a general strategy to convert the json based message to the nodes and relationships. even like in the end we have a Graph DB output/input and then graphDB engine is specified as part of the config

peterclemenko commented 2 months ago

Neo4j is the standard right now, gql is coming but not there yet, it would be wise to implement neo4j for now and gql down the line

rockwotj commented 1 month ago

I'm planning on working on this one this week. Just an output to start. Might be fun to have a processor for queries too.