moj-analytical-services / s3tools

Interact with files in s3 on the Analytical Platform
8 stars 2 forks source link

Would it be better if the default CSV reader was fread? #47

Open markpurver opened 4 years ago

markpurver commented 4 years ago

This isn't a major issue as fread can be used within read_using, but fread is faster than read.csv and copes well with untidy CSVs (e.g. where the first row has fewer columns than the rest, which unfortunately occurs in downloads of SOP reports).

Both fread and read_csv are fast and apply similar default options to read_excel (e.g. trimming white space), but read_csv doesn't cope so well with untidy CSVs.

I realise either way it creates a dependency on another package, which may be better avoided.