CleverCSV is a Python package for handling messy CSV files. It provides a drop-in replacement for the builtin CSV module with improved dialect detection, and comes with a handy command line application for working with CSV files.
This PR brings a redesign of the consistency score calculation to allow for caching of the type detection results. This reduces the median runtime by 64% compared to the current master branch (computed similarly as in #92). The average runtime on our test set is reduced by ~32% compared to the current master branch. It is likely that further performance improvements are possible.
Compared to v0.7.6, CleverCSV is now ~52% faster on average, and median runtime is reduced by 68%.
This PR brings a redesign of the consistency score calculation to allow for caching of the type detection results. This reduces the median runtime by 64% compared to the current master branch (computed similarly as in #92). The average runtime on our test set is reduced by ~32% compared to the current master branch. It is likely that further performance improvements are possible.
Compared to v0.7.6, CleverCSV is now ~52% faster on average, and median runtime is reduced by 68%.