Open CameronBieganek opened 4 months ago
Note that scan_csv
works, like this:
data = pl.scan_csv("test.csv", schema=schema)
...where the file and the schema
dictionary are the same as above. I'm guessing the error with read_csv
is happening because the column order in the schema does not match the column order in the CSV? Normally I expect the order of entries in a dictionary to be immaterial, although technically as of Python 3.6 the built-in dictionary preserves insertion order.
I have a very similar issue open already. Basically this comes down to very poor error messages when the schema
argument is involved. Not to mention, the docstring entry for schema
could be more explicit about the requirements: e.g. order of entries in the dictionary must match the order of the columns.
That is odd.
Just a visualization of how schema
is treated differently in read
and scan
:
import tempfile
import polars as pl
f = tempfile.NamedTemporaryFile()
f.write(b"""
A,B
1,2
""".strip())
f.seek(0)
pl.read_csv(f.name, schema={"B": pl.String, "A": pl.Int32})
# shape: (1, 2)
# ┌─────┬─────┐
# │ B ┆ A │
# │ --- ┆ --- │
# │ str ┆ i32 │
# ╞═════╪═════╡
# │ 1 ┆ 2 │
# └─────┴─────┘
pl.scan_csv(f.name, schema={"B": pl.String, "A": pl.Int32}).collect()
# shape: (1, 2)
# ┌─────┬─────┐
# │ A ┆ B │
# │ --- ┆ --- │
# │ i32 ┆ str │
# ╞═════╪═════╡
# │ 1 ┆ 2 │
# └─────┴─────┘
[Update]: - It seems https://github.com/pola-rs/polars/issues/11723 contains a mention of it.
Found in the redesign issue:
Checks
Reproducible example
Use the following CSV file:
And the following Python script:
Output:
Installed versions