CSV data with double quotes fails

apache / datafusion

Apache DataFusion SQL Query Engine

https://datafusion.apache.org/

Apache License 2.0

6.06k stars 1.15k forks source link

CSV data with double quotes fails #439

Open SINHASantos opened 3 years ago

SINHASantos commented 3 years ago

Describe the bug CSV data with double quotes fails

To Reproduce Pass CSV data with double quotes

Expected behavior Datafusion shall be able be able to parse the csv data with double or sing quotes example:

"\"Hacksaw\" Jim Duggan";1987"Bret \"Hit Man\" Hart";1984

Additional context rust-csv is able to doc it correctly, where as arrow-datafusion throws error to read the csv data with doube quotes or single quotes

alamb commented 3 years ago

Perhaps you can specify the options you want via the CsvReadOptions argument: https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/execution/context.rs#L274