CSV can't parse null value for non-string type (i32, i64, float) - Githubissues

apache / datafusion

Apache DataFusion SQL Query Engine

https://datafusion.apache.org/

Apache License 2.0

6.13k stars 1.16k forks source link

CSV can't parse null value for non-string type (i32, i64, float) #12904

Open jayzhan211 opened 4 days ago

jayzhan211 commented 4 days ago

Describe the bug

As title, I would like to store null value in the csv table but couldn't.

To Reproduce

Write a csv file to datafusion/core/tests/data/data.csv with

c1,c2
NULL,'a'

statement ok
CREATE EXTERNAL TABLE t (
c1 INT,
c2 VARCHAR,
)
STORED AS CSV
LOCATION '../core/tests/data/data.csv'
OPTIONS ('format.has_header' 'true');

query error DataFusion error: Arrow error: Parser error: Error while parsing value NULL for column 0 at line 1
select * from t;

Expected behavior

Successfully store nulls for non-string type

Additional context

String is fine

Omega359 commented 3 days ago

I think this may be something best handled in the arrow-csv crate