Support multiple files per-entity/relationship type

typedb-osi / typedb-loader

TypeDB Loader - Data Migration Tool for TypeDB

https://github.com/typedb-osi/typedb-loader

Apache License 2.0

58 stars 17 forks source link

Support multiple files per-entity/relationship type #19

Closed suciokhan closed 2 years ago

suciokhan commented 3 years ago

I have hundreds of files for a dataset with ~100 million records where in many cases a single entity type's records are spread across 20 different files. From the dataConfig.json example, it appears I can only associate a single .csv file to a given entity/relationship type at a time. It would be quite nice to be able to configure multiple dataPaths for all the different files I have for a single entity type rather than have to have 20 different config files that I have to run separately.

suciokhan commented 3 years ago

Perhaps allow something like "dataPath": ['/list/of/paths1',/list/of/paths2',...] ?

hkuich commented 3 years ago

thanks for this - will implement this as soon as 2.0 is stabilized! Will let you know!

hkuich commented 3 years ago

implementation finished - testing thoroughly now...

hkuich commented 2 years ago

fixed an released @suciokhan