General idea is to concurrently import metadata files into separate SQLite dbs, then ATTACH them to export merged metadata.
I whipped up a working prototype and pushed it to trs/merge/concurrent-import. On a trivial two file, three record input set, this alone cuts off 1.5s of wall clock time, because of the slow slow slow augur start up time being paid concurrently instead of serially for the two input files passed thru augur read-file.
General idea is to concurrently import metadata files into separate SQLite dbs, then
ATTACHthem to export merged metadata.I whipped up a working prototype and pushed it to
trs/merge/concurrent-import. On a trivial two file, three record input set, this alone cuts off 1.5s of wall clock time, because of the slow slow slowaugurstart up time being paid concurrently instead of serially for the two input files passed thruaugur read-file.