This pull requests adds a SQLite3 intermediate layer. Concepts are first imported from spreadsheet to the SQLite3 database, and then exported from the database to YAMLs.
Two CLI commands are introduced: xlsx2db and db2yaml. The xlsx2yaml which existed before is preserved, though now it uses SQL for intermediate data representation.
Motivation
I need some data analysis tool, and for many reasons SQL suits me better than spreadsheets.
Now I can convert the spreadsheet to SQL, and then use that SQL source for testing concepts processing. This saves a considerable amount of time at each iteration, because parsing spreadsheets is really slow.
Thanks to SQLite3 console, it's easy to cut an arbitrary subset of input data to work on.
Database constraints already helped in anomaly detection, e.g. #93.
Performance considerations
The intermediate layer adds some overhead when converting from XLS to YAMLs, but it's insignificant (2% slower on my computer).
SQLite3 increases memory usage, but the database size is just about 50 MB.
Final notes
This pull request adds SQLite3 as a dependency. I can make it optional if someone is unhappy with that.
The result of xlsx2yaml is slightly different in case of duplicated rows (see #93). It used to take the last row for given term ID and language pair, but now it takes the first one.
This pull requests adds a SQLite3 intermediate layer. Concepts are first imported from spreadsheet to the SQLite3 database, and then exported from the database to YAMLs.
Two CLI commands are introduced:
xlsx2db
anddb2yaml
. Thexlsx2yaml
which existed before is preserved, though now it uses SQL for intermediate data representation.Motivation
Performance considerations
Final notes
xlsx2yaml
is slightly different in case of duplicated rows (see #93). It used to take the last row for given term ID and language pair, but now it takes the first one.