pola-rs / polars

Dataframes powered by a multithreaded, vectorized query engine, written in Rust
https://docs.pola.rs
Other
27.72k stars 1.69k forks source link

ComputeError when writing `Decimal` type to csv using `write_csv` #14773

Open JulianCologne opened 4 months ago

JulianCologne commented 4 months ago

Checks

Reproducible example

(
    pl.DataFrame({"x": ["0.1", "0.2"]})
    .with_columns(pl.col("x").cast(pl.Decimal))
    .write_csv("decimal.csv")
)

# shape: (2, 1)
# ┌──────────────┐
# │ x            │
# │ ---          │
# │ decimal[*,1] │
# ╞══════════════╡
# │ 0.1          │
# │ 0.2          │
# └──────────────┘

Log output

ComputeError: datatype 0.1 cannot be written to csv

Issue description

DataFrames containing columns of Decimal type cannot be written to csv

Expected behavior

It is possible to display/print a dataframe so it should be possitlbe to also write Decimal to csv.

Installed versions

``` 0.20.11 ```
flisky commented 3 months ago

I think #14209 fixes this issue.

MarceloCFerraz commented 1 month ago

This got me through this issue. Maybe it can help anyone that has a Decimal column and is trying to save the DataFrame to a csv file:

import polars as pl

copy = df

# Get data types of columns
column_dtypes = copy.dtypes

for i in range(len(column_dtypes)):
    c_type = column_dtypes[i]

    if c_type.is_decimal():
        column = df.columns[i]

        print(f'Converting {column} to Float')

        copy = copy.with_columns(copy[column].cast(pl.Float64))

copy.write_csv("test.csv")
JulianCologne commented 1 month ago

@MarceloCFerraz the same can be done in polars in just one line

df.with_columns(pl.col(pl.Decimal).cast(pl.Float64))

However, this is a workaround for some use cases but not an ideal solution 😄