Open angelosantos4 opened 1 year ago
I agree that this could be handy. Question marks around whether this is required in an ETL framework. Definitely willing to take PRs on this.
One major challenge is going to be around retaining the abstraction behind graph databases.
Is your feature request related to a problem? Please describe. When we ingest datatypes from different sources, we may run into the issue where we ingest data from an old source and one from a recent source with different properties. The current implementation builds a merge query which simply overwrites the properties on matched nodes based on which record was ingested first. I would like for there to be a way to conditionally change properties on a node.
Describe the solution you'd like When I create an interpretation within my pipeline, I would like to declare the following:
This would then modify the merge query which currently performs the following for source nodes:
I would like it to create the following:
Where condition in our case would be
(date_created > node.date_created)
Describe alternatives you've considered The alternative I can perform to ensure the recency of my data is I can schedule my pipelines such that the recent data comes in after the old data. In my pipeline, I can create an interpreter that makes a call to the database to get the value, then conditionally write to the database (this takes too long.)
Additional context