Closed proever closed 2 months ago
partitioning only on "ticker" causes the issue too, I just checked
pq.ParquetDataset(dataset_path).read().to_pandas()
works fine as well
this seems fixed in 1.0, at least with the following changes:
from datetime import datetime
from pathlib import Path
import polars as pl
dataset_path = Path("./test_dataset")
dataset_path.mkdir(exist_ok=True)
test_df = (
pl.DataFrame(
{
"timestamp": [datetime(2021, 1, 1), datetime(2022, 2, 1)],
"data": [1, 2],
"ticker": ["AAPL", "TRUE"],
}
)
.with_columns(pl.col("ticker").cast(pl.Enum(["AAPL", "TRUE"])))
.with_columns(
pl.col("timestamp").dt.year().alias("year"), pl.col("timestamp").dt.month().alias("month")
)
)
test_df.write_parquet(
dataset_path, use_pyarrow=True, pyarrow_options={"partition_cols": ["ticker", "year", "month"]}
)
df_in = pl.scan_parquet(dataset_path, hive_partitioning=True)
print(df_in.filter(pl.col("ticker") == "AAPL").collect())
Checks
Reproducible example
Log output
Issue description
I have a large dataset of stock data, which includes ticker names. One of those tickers is "TRUE" (apparently it's the company TrueCar). I am trying to partition the dataset based on, among other things, the ticker. This seems to work fine, but when I try to load the dataset back in with
scan_csv
, polars complains about parsing the "TRUE" partition (see above).Changing the value of "TRUE" to "TEST" (in the data and the Enum) fixes the issue. Changing it back to "FALSE" causes the issue to occur again, which leads me to believe there is some logic being applied to attempt to parse boolean values, causing the issue.
Expected behavior
code above should run with no errors, as it does when "TRUE" is replaced with "TEST" (for example)
Installed versions