awslabs / amazon-neptune-tools

Tools and utilities to enable loading data and building graph applications with Amazon Neptune.
Apache License 2.0
297 stars 151 forks source link

[glue-neptune] Add batching within each Gremlin Traversal for inserts/upserts to improve throughput #33

Closed triggan closed 4 years ago

triggan commented 5 years ago

The current implementation inserts each vertex and edge one at a time along with their properties. Based on Neptune best practices [1], each Gremlin traversal for inserts/upserts should apply 50-100 objects (an object being a unique vertex, unique edge, and any vertex/edge property) using a single Gremlin traversal.

[1] https://github.com/aws-samples/aws-dbs-refarch-graph/tree/master/src/writing-from-amazon-kinesis-data-streams

beebs-systap commented 4 years ago

Closed in https://github.com/awslabs/amazon-neptune-tools/pull/54