Error when the result of joining two `DataFrame`s is empty, and one of the `DataFrame`s contains an array column

pola-rs / polars

Dataframes powered by a multithreaded, vectorized query engine, written in Rust

Other

30.22k stars 1.95k forks source link

Checks

[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl

a = pl.DataFrame([
    pl.Series("x", [1, 2, 3]),
    pl.Series("y", [[1, 2, 3], [4, 5, 6], [7, 8, 9]]).list.to_array(3),
])

b = pl.DataFrame([
    pl.Series("x", [0]),
])

print(a.join(b, "x"))

Log output

join parallel: true
thread 'polars-1' panicked at /home/runner/work/polars/polars/crates/polars-arrow/src/array/static_array_collect.rs:929:18:
called `Result::unwrap()` on an `Err` value: InvalidOperation(ErrString("concat requires input of at least one array"))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "/home/andrei/example.py", line 13, in <module>
    print(a.join(b, "x"))
  File "/home/andrei/.local/lib/python3.10/site-packages/polars/dataframe/frame.py", line 6392, in join
    self.lazy()
  File "/home/andrei/.local/lib/python3.10/site-packages/polars/lazyframe/frame.py", line 1943, in collect
    return wrap_df(ldf.collect())
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: InvalidOperation(ErrString("concat requires input of at least one array"))

Issue description

Tested on Ubuntu 22.04.4 LTS.

Same error happens if I swap a and b in the join.

import polars as pl

a = pl.DataFrame([
    pl.Series("x", [1, 2, 3]),
    pl.Series("y", [[1, 2, 3], [4, 5, 6], [7, 8, 9]]).list.to_array(3),
])

b = pl.DataFrame([
    pl.Series("x", [0]),
])

print(b.join(a, "x")) # swapped a with b

If I use a list instead of an array, it works as expected.

import polars as pl

a = pl.DataFrame([
    pl.Series("x", [1, 2, 3]),
    pl.Series("y", [[1, 2, 3], [4, 5, 6], [7, 8, 9]]), # removed cast to array
])

b = pl.DataFrame([
    pl.Series("x", [0]),
])

print(a.join(b, "x"))

If the result of joining a and b is not empty, then it also works as expected.

import polars as pl

a = pl.DataFrame([
    pl.Series("x", [1, 2, 3]),
    pl.Series("y", [[1, 2, 3], [4, 5, 6], [7, 8, 9]]).list.to_array(3),
])

b = pl.DataFrame([
    pl.Series("x", [0, 1, 2]), # added some values
])

print(a.join(b, "x"))

Expected behavior

The join should output an empty DataFrame. For the provided example, the output should look something like this:

┌─────┬───────────────┐
│ x   ┆ y             │
│ --- ┆ ---           │
│ i64 ┆ array[i64, 3] │
╞═════╪═══════════════╡
└─────┴───────────────┘

Installed versions

``` --------Version info--------- Polars: 0.20.18 Index type: UInt32 Platform: Linux-6.5.0-26-generic-x86_64-with-glibc2.35 Python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] ----Optional dependencies---- adbc_driver_manager: cloudpickle: 3.0.0 connectorx: deltalake: fastexcel: fsspec: 2023.12.2 gevent: hvplot: matplotlib: 3.5.1 nest_asyncio: numpy: 1.21.5 openpyxl: 3.0.9 pandas: 1.3.5 pyarrow: pydantic: pyiceberg: pyxlsb: sqlalchemy: xlsx2csv: xlsxwriter: ```

# works ( pl.DataFrame({ "id": pl.Series([], dtype=pl.Utf8()), "val": pl.Series([], dtype=pl.List(pl.Float32)), }) .group_by(pl.col("id")) .first() ) # fails ( pl.DataFrame({ "id": pl.Series([], dtype=pl.Utf8()), "val": pl.Series([], dtype=pl.Array(pl.Float32, 20), )}) .group_by(pl.col("id")) .first() ) # exception: `PanicException: called `Result::unwrap()` on an `Err` value: InvalidOperation(ErrString("concat requires input of at least one array"))`.

pola-rs / polars