We need an ingest tool that allows us to input local files to the DB.
This PR addresses #28 (but WIP).
The design of the DB structure disallows updates concurrently, which limits how efficient the data ingestion is.
For now, I submit this PR that provides an initial glimpse of how the pipeline could be constructed.
The operations in the pipeline themselves will have to be changed after fixing #29.
juagargi
commented
1 year ago
We need an ingest tool that allows us to input local files to the DB.
This PR addresses #28 (but WIP). The design of the DB structure disallows updates concurrently, which limits how efficient the data ingestion is. For now, I submit this PR that provides an initial glimpse of how the pipeline could be constructed. The operations in the pipeline themselves will have to be changed after fixing #29.
TODO:
This change is