apache / datafusion

Apache DataFusion SQL Query Engine
https://datafusion.apache.org/
Apache License 2.0
6.37k stars 1.2k forks source link

CSV can't parse null value for non-string type (i32, i64, float) #12904

Open jayzhan211 opened 1 month ago

jayzhan211 commented 1 month ago

Describe the bug

As title, I would like to store null value in the csv table but couldn't.

To Reproduce

Write a csv file to datafusion/core/tests/data/data.csv with

c1,c2
NULL,'a'
statement ok
CREATE EXTERNAL TABLE t (
c1 INT,
c2 VARCHAR,
)
STORED AS CSV
LOCATION '../core/tests/data/data.csv'
OPTIONS ('format.has_header' 'true');

query error DataFusion error: Arrow error: Parser error: Error while parsing value NULL for column 0 at line 1
select * from t;

Expected behavior

Successfully store nulls for non-string type

Additional context

String is fine

Omega359 commented 1 month ago

I think this may be something best handled in the arrow-csv crate

dhegberg commented 3 weeks ago

take