search

intuit / superglue

Superglue is a lineage-tracking tool built to help visualize the propagation of data through complex pipelines composed of tables, jobs and reports.
Apache License 2.0
155 stars 37 forks source link

Graph Model for SuperGlue Lineage Services #11

Open akumarb2010 opened 4 years ago

akumarb2010 commented 4 years ago

Is your feature request related to a problem? Please describe. Currently, superGlue is using RDBMS for graph modeling and lineage services will construct the Graph from metadata. The challenges with this approach is as below.

  1. Handling full property Graph features will become nightmare and not scalable
  2. To handling column lineage will become more complex with current model
  3. Graph property based index is not straight forward.

To handle all the above issues, at Intuit, we have started working Graph model and integration with Neo4j.

Describe the solution you'd like Coming up with Property Graph schema, integrating SuperGlue with Neo4j.

Describe alternatives you've considered We will be working on end-end design document with all the details on various approaches

Additional context NA

smandalika1 commented 4 years ago

@akumarb2010 - Is this open for contribution?