[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of Polars.
Reproducible example
master = pl.scan_parquet(out_path, low_memory=True).select(columns=columns)
paths = [
p for p in activity_embeddings_path.parent.listdir() if str(p)[-1].isnumeric()
]
for p in paths:
df = pl.scan_parquet(p, low_memory=True).select(columns=columns)
master = pl.concat([master, df])
master = master.unique(subset=key_column)
master.sink_parquet("/tmp/xx.parquet")
p.unlink()
Log output
No response
Issue description
POLARS PREFETCH_SIZE: 24
thread '<unnamed>' panicked at /home/runner/work/polars/polars/crates/polars-pipe/src/pipeline/convert.rs:409:88:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "/home/torstein/code/nysno/nysno/embedding.py", line 531, in <module>
merge_temp()
File "/home/torstein/code/nysno/nysno/embedding.py", line 513, in merge_temp
master.sink_parquet("/tmp/xx.parquet")
File "/home/torstein/code/nysno/.venv/lib/python3.12/site-packages/polars/_utils/unstable.py", line 58, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/torstein/code/nysno/.venv/lib/python3.12/site-packages/polars/lazyframe/frame.py", line 2385, in sink_parquet
return lf.sink_parquet(
^^^^^^^^^^^^^^^^
pyo3_runtime.PanicException: called `Option::unwrap()` on a `None` value
Checks
Reproducible example
Log output
No response
Issue description
Expected behavior
Should concat parquets
Installed versions