Optimize the normal form detection

alan-turing-institute / CleverCSV

CleverCSV is a Python package for handling messy CSV files. It provides a drop-in replacement for the builtin CSV module with improved dialect detection, and comes with a handy command line application for working with CSV files.

https://clevercsv.readthedocs.io

MIT License

1.24k stars 70 forks source link

Optimize the normal form detection #123

Closed no23reason closed 3 months ago

no23reason commented 3 months ago

Aimed at avoiding as much full file scans as possible, this PR should bring improved performance of the normal form detection.

Steps taken (there are more details in the individual commits):

optimize the even_rows logic so that it exits as soon as possible instead of going through the whole file
avoid repeated file splitting by splitting the file once and passing the split rows to the individual is_form_x functions

no23reason commented 3 months ago

Sorry for the failed build, I amended the formatting issues.

GjjvdBurg commented 3 months ago

Thanks for opening this PR @no23reason! Looks like there are just a few build failures to iron out, but other than that it looks good

GjjvdBurg commented 3 months ago

Thanks again @no23reason!

no23reason commented 3 months ago

Thank you, especially for the patience with the mistakes I should have caught faster :)