search

typedb-osi / typedb-loader

TypeDB Loader - Data Migration Tool for TypeDB
https://github.com/typedb-osi/typedb-loader
Apache License 2.0
58 stars 17 forks source link

Remove row-groups batching layer #34

Closed flyingsilverfin closed 2 years ago

flyingsilverfin commented 2 years ago

Goal

We simplify the ingestion pipeline by removing a batching layer that batches collections of rows.

Description

Previously, we had a queue of list of list of rows. This represented: queue of batches of sets of rows, where sets of rows are to be committed in one transaction, and the batch holds a series data for transactions to commit. This middle layer of batching was incorporated to to avoid too much contention on the queue that feeds the transaction threads.

However, testing shows this has negligible impact and we can simplify the architecture, reduce memory consumption, and enable later fatures by removing this batching layer.

flyingsilverfin commented 2 years ago

Closed as a consequence that one of the things it may simplify is https://github.com/bayer-science-for-a-better-life/grami/issues/35 , which has a lot of other more fundamental blockers.