I recently started using csv-to-neptune-bulk-format after finding that my custom solution (using Gremlin query strings by row) didn't deliver the performance benefits of a bulk loader as promised. One improvement I believe could enhance the converter tool is the capability to utilize existing vertex IDs from Neptune. This would allow for creating edges based on data from both ends of the edge solving late arriving data issues.
For instance, consider the scenario with Consumer and Orders data. The Consumer dataset contains consumer information but lacks order details. As a result, we cannot define "Consumer_to_Order" edges because we don't have visibility into which Orders each Consumer has in Neptune. These edges can only be established once we receive the Order data.
I am planning to implement this enhancement as a fork of the tool. Do you think it would be beneficial to merge this back into the main repository once completed?
Hello,
I recently started using csv-to-neptune-bulk-format after finding that my custom solution (using Gremlin query strings by row) didn't deliver the performance benefits of a bulk loader as promised. One improvement I believe could enhance the converter tool is the capability to utilize existing vertex IDs from Neptune. This would allow for creating edges based on data from both ends of the edge solving late arriving data issues.
For instance, consider the scenario with Consumer and Orders data. The Consumer dataset contains consumer information but lacks order details. As a result, we cannot define "Consumer_to_Order" edges because we don't have visibility into which Orders each Consumer has in Neptune. These edges can only be established once we receive the Order data.
I am planning to implement this enhancement as a fork of the tool. Do you think it would be beneficial to merge this back into the main repository once completed?
Thanks in advance.