search

pola-rs / polars

Dataframes powered by a multithreaded, vectorized query engine, written in Rust
https://docs.pola.rs
Other
29.28k stars 1.85k forks source link

Panic with "expected arrays of the same length" #18673

Open kszlim opened 6 days ago

kszlim commented 6 days ago

Checks

Reproducible example

import polars as pl

df = pl.DataFrame([
    pl.Series('x', [{ 'a': 1 }]),
    pl.Series('b', [None], dtype=pl.Boolean),
])

df = df.vstack(df)

result = df.select(
    pl.when(pl.col.b).then(pl.first("x").alias("x")).otherwise(pl.first("x"))
)

panic.txt

Log output

thread 'polars-21' panicked at crates/polars-core/src/utils/mod.rs:914:5:
expected arrays of the same length
stack backtrace:
   0:     0x7f5ba816973d - std::backtrace_rs::backtrace::libunwind::trace::h91845c451f8de049
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1:     0x7f5ba816973d - std::backtrace_rs::backtrace::trace_unsynchronized::hba51bd6b35f4c19c
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7f5ba816973d - std::sys::backtrace::_print_fmt::h5df6ae8419e5a7d0
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/std/src/sys/backtrace.rs:66:9
   3:     0x7f5ba816973d - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h08c68512b2311366
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/std/src/sys/backtrace.rs:39:26
   4:     0x7f5ba819771b - core::fmt::rt::Argument::fmt::h11296e5540fbe1b9
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/core/src/fmt/rt.rs:173:76
   5:     0x7f5ba819771b - core::fmt::write::h6e3f8345ff37039b
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/core/src/fmt/mod.rs:1178:21
   6:     0x7f5ba8165633 - std::io::Write::write_fmt::hc6823578ffc00004
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/std/src/io/mod.rs:1823:15
   7:     0x7f5ba816a882 - std::sys::backtrace::BacktraceLock::print::hdba6c7eedfb8a6b8
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/std/src/sys/backtrace.rs:42:9
   8:     0x7f5ba816a882 - std::panicking::default_hook::{{closure}}::hee63056ca50632bf
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/std/src/panicking.rs:266:22
   9:     0x7f5ba816a4ee - std::panicking::default_hook::hf7cfdbe4468ff824
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/std/src/panicking.rs:293:9
  10:     0x7f5ba816b10f - std::panicking::rust_panic_with_hook::hdfce2b8bd7fc4cbd
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/std/src/panicking.rs:797:13
  11:     0x7f5ba816adc3 - std::panicking::begin_panic_handler::{{closure}}::h1fea56f9bbc92609
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/std/src/panicking.rs:664:13
  12:     0x7f5ba8169c29 - std::sys::backtrace::__rust_end_short_backtrace::h60ce9d111c920856
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/std/src/sys/backtrace.rs:170:18
  13:     0x7f5ba816aa84 - rust_begin_unwind
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/std/src/panicking.rs:662:5
  14:     0x7f5ba2b5f933 - core::panicking::panic_fmt::hca2ff70b38236a4a
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/core/src/panicking.rs:74:14
  15:     0x7f5ba6c93ed0 - polars_core::utils::align_chunks_ternary::hf69c1b2f0c522a9a
                               at /local/polars/crates/polars-core/src/utils/mod.rs:914:5
  16:     0x7f5ba6e394ad - polars_core::chunked_array::ops::zip::<impl polars_core::chunked_array::ops::ChunkZip<polars_core::datatypes::StructType> for polars_core::chunked_array::ChunkedArray<polars_core::datatypes::StructType>>::zip_with::h2fa4a4c1f797b341
                               at /local/polars/crates/polars-core/src/chunked_array/ops/zip.rs:217:28
  17:     0x7f5ba73c5f91 - polars_core::series::implementations::struct_::<impl polars_core::series::series_trait::private::PrivateSeries for polars_core::series::implementations::SeriesWrap<polars_core::chunked_array::ChunkedArray<polars_core::datatypes::StructType>>>::zip_with_same_type::h920c0c7dc45c471f
                               at /local/polars/crates/polars-core/src/series/implementations/struct_.rs:53:9
  18:     0x7f5ba68515e7 - polars_core::series::Series::zip_with::ha5dd8f73c27ab1c5
                               at /local/polars/crates/polars-core/src/series/mod.rs:592:9
  19:     0x7f5ba4740bc7 - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::h3cb4d5d7821b49b6
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:167:23
  20:     0x7f5ba478cb3d - <polars_expr::expressions::cast::CastExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::h8d034e6044d8baca
                               at /local/polars/crates/polars-expr/src/expressions/cast.rs:37:22
  21:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::{{closure}}::ha9b303dc863d7146
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:112:27
  22:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::h3cb4d5d7821b49b6
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:116:39
  23:     0x7f5ba478cb3d - <polars_expr::expressions::cast::CastExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::h8d034e6044d8baca
                               at /local/polars/crates/polars-expr/src/expressions/cast.rs:37:22
  24:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::{{closure}}::ha9b303dc863d7146
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:112:27
  25:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::h3cb4d5d7821b49b6
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:116:39
  26:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::{{closure}}::ha9b303dc863d7146
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:112:27
  27:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::h3cb4d5d7821b49b6
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:116:39
  28:     0x7f5ba478cb3d - <polars_expr::expressions::cast::CastExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::h8d034e6044d8baca
                               at /local/polars/crates/polars-expr/src/expressions/cast.rs:37:22
  29:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::{{closure}}::ha9b303dc863d7146
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:112:27
  30:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::h3cb4d5d7821b49b6
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:116:39
  31:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::{{closure}}::ha9b303dc863d7146
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:112:27
  32:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::h3cb4d5d7821b49b6
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:116:39
  33:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::{{closure}}::ha9b303dc863d7146
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:112:27
  34:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::h3cb4d5d7821b49b6
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:116:39
  35:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::{{closure}}::ha9b303dc863d7146
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:112:27
  36:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::h3cb4d5d7821b49b6
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:116:39
  37:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::{{closure}}::ha9b303dc863d7146
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:112:27
  38:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::h3cb4d5d7821b49b6
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:116:39
  39:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::{{closure}}::ha9b303dc863d7146
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:112:27
  40:     0x7f5ba473fa1a - <polars_expr::expressions::ternary::TernaryExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::h3cb4d5d7821b49b6
                               at /local/polars/crates/polars-expr/src/expressions/ternary.rs:116:39
  41:     0x7f5ba47768a1 - <polars_expr::expressions::alias::AliasExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::h64f1869c8d62572e
                               at /local/polars/crates/polars-expr/src/expressions/alias.rs:43:22
  42:     0x7f5ba47cf015 - <polars_expr::expressions::apply::ApplyExpr as polars_expr::expressions::PhysicalExpr>::evaluate_on_groups::h8b2ee973a91ac8ba
                               at /local/polars/crates/polars-expr/src/expressions/apply.rs:351:26
  43:     0x7f5ba473213e - polars_expr::expressions::window::WindowExpr::run_aggregation::h1699b3346c614b6c
                               at /local/polars/crates/polars-expr/src/expressions/window.rs:201:18
  44:     0x7f5ba473213e - <polars_expr::expressions::window::WindowExpr as polars_expr::expressions::PhysicalExpr>::evaluate::h61fac24f5bd29f4a
                               at /local/polars/crates/polars-expr/src/expressions/window.rs:516:22
  45:     0x7f5ba4776777 - <polars_expr::expressions::alias::AliasExpr as polars_expr::expressions::PhysicalExpr>::evaluate::h4b1cb61c684542e1
                               at /local/polars/crates/polars-expr/src/expressions/alias.rs:32:22
  46:     0x7f5ba46c0c0d - polars_mem_engine::executors::projection_utils::window_evaluate::{{closure}}::{{closure}}::hbe346096b7a777ed
                               at /local/polars/crates/polars-mem-engine/src/executors/projection_utils.rs:88:29
  47:     0x7f5ba46c0c0d - core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut::h32796deb75b6fa68
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/core/src/ops/function.rs:272:13
  48:     0x7f5ba464099b - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once::h4e09029778cc787e
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/core/src/ops/function.rs:305:13
  49:     0x7f5ba464099b - core::option::Option<T>::map::h30acfe0b963ed37f
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/core/src/option.rs:1105:29
  50:     0x7f5ba464099b - <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next::h783b84e33efc130b
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/core/src/iter/adapters/map.rs:107:26
  51:     0x7f5ba464099b - <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next::h59778005554ac7cd
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/core/src/iter/adapters/map.rs:107:9
  52:     0x7f5ba464099b - <core::iter::adapters::take_while::TakeWhile<I,P> as core::iter::traits::iterator::Iterator>::next::h821365e4a9a632ab
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/core/src/iter/adapters/take_while.rs:48:21
  53:     0x7f5ba464099b - <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next::h0b163ba83b975255
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/core/src/iter/adapters/map.rs:107:9
  54:     0x7f5ba464099b - alloc::vec::Vec<T,A>::extend_desugared::hd981ec0ba8fdce6d
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/alloc/src/vec/mod.rs:3089:35
  55:     0x7f5ba46c7018 - <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend::h9b60b7f476e89310
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/alloc/src/vec/spec_extend.rs:17:9
  56:     0x7f5ba46c7018 - <alloc::vec::Vec<T,A> as core::iter::traits::collect::Extend<T>>::extend::h3546c1235eb14c91
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/alloc/src/vec/mod.rs:3053:9
  57:     0x7f5ba46c7018 - <rayon::iter::extend::ListVecFolder<T> as rayon::iter::plumbing::Folder<T>>::consume_iter::h9350be75c7b473d5
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/extend.rs:130:9
  58:     0x7f5ba46c7018 - <rayon::iter::while_some::WhileSomeFolder<C> as rayon::iter::plumbing::Folder<core::option::Option<T>>>::consume_iter::h5d71de207a3f2d3f
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/while_some.rs:139:21
  59:     0x7f5ba46c7018 - <rayon::iter::map::MapFolder<C,F> as rayon::iter::plumbing::Folder<T>>::consume_iter::hf185ec762b5de39f
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/map.rs:248:21
  60:     0x7f5ba46c7018 - <rayon::iter::map::MapFolder<C,F> as rayon::iter::plumbing::Folder<T>>::consume_iter::hb55d2b8e78a54839
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/map.rs:248:21
  61:     0x7f5ba46c7018 - <rayon::iter::map::MapFolder<C,F> as rayon::iter::plumbing::Folder<T>>::consume_iter::hddc9c35767f55de8
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/map.rs:248:21
  62:     0x7f5ba46c7018 - <hashbrown::external_trait_impls::rayon::raw::ParIterProducer<T> as rayon::iter::plumbing::UnindexedProducer>::fold_with::h338440b6caded998
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/hashbrown-0.14.5/src/external_trait_impls/rayon/raw.rs:73:9
  63:     0x7f5ba46c7018 - rayon::iter::plumbing::bridge_unindexed_producer_consumer::hd15867ff24c364ad
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/plumbing/mod.rs:478:33
  64:     0x7f5ba46423be - rayon::iter::plumbing::bridge_unindexed::h68dedfa824321e09
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/plumbing/mod.rs:451:5
  65:     0x7f5ba46423be - <hashbrown::external_trait_impls::rayon::raw::RawParIter<T> as rayon::iter::ParallelIterator>::drive_unindexed::h38b0b6cbb264d7ed
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/hashbrown-0.14.5/src/external_trait_impls/rayon/raw.rs:48:9
  66:     0x7f5ba46423be - <rayon::iter::map::Map<I,F> as rayon::iter::ParallelIterator>::drive_unindexed::h3a01860186cbb335
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/map.rs:49:9
  67:     0x7f5ba46423be - <hashbrown::external_trait_impls::rayon::map::ParIter<K,V> as rayon::iter::ParallelIterator>::drive_unindexed::h430213b0b598aade
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/hashbrown-0.14.5/src/external_trait_impls/rayon/map.rs:34:9
  68:     0x7f5ba46423be - <rayon::iter::map::Map<I,F> as rayon::iter::ParallelIterator>::drive_unindexed::hcd0434391ad38833
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/map.rs:49:9
  69:     0x7f5ba46423be - <rayon::iter::map::Map<I,F> as rayon::iter::ParallelIterator>::drive_unindexed::h299d7329fd048a88
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/map.rs:49:9
  70:     0x7f5ba46423be - <rayon::iter::while_some::WhileSome<I> as rayon::iter::ParallelIterator>::drive_unindexed::hf6f146d0b3cebf72
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/while_some.rs:44:9
  71:     0x7f5ba46423be - rayon::iter::extend::<impl rayon::iter::ParallelExtend<T> for alloc::vec::Vec<T>>::par_extend::h1cc6fa0284ee006f
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/extend.rs:588:28
  72:     0x7f5ba469ceb9 - rayon::iter::from_par_iter::collect_extended::h6fee0c89e1d3d8bb
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/from_par_iter.rs:20:5
  73:     0x7f5ba469ceb9 - rayon::iter::from_par_iter::<impl rayon::iter::FromParallelIterator<T> for alloc::vec::Vec<T>>::from_par_iter::h5043b0cca99d6c01
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/from_par_iter.rs:33:9
  74:     0x7f5ba469ceb9 - rayon::iter::ParallelIterator::collect::h855a53ec983fcc05
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/mod.rs:2069:9
  75:     0x7f5ba469ceb9 - rayon::result::<impl rayon::iter::FromParallelIterator<core::result::Result<T,E>> for core::result::Result<C,E>>::from_par_iter::ha6d52365b70c720d
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/result.rs:121:26
  76:     0x7f5ba4689abf - rayon::iter::ParallelIterator::collect::hb31fd7c65d35abe3
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/mod.rs:2069:9
  77:     0x7f5ba4689abf - polars_mem_engine::executors::projection_utils::window_evaluate::{{closure}}::h21c7d05a506964f2
                               at /local/polars/crates/polars-mem-engine/src/executors/projection_utils.rs:55:9
  78:     0x7f5ba4689abf - rayon_core::thread_pool::ThreadPool::install::{{closure}}::h20b42d78adfade15
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/thread_pool/mod.rs:147:40
  79:     0x7f5ba4689abf - rayon_core::registry::Registry::in_worker::h4890c3db5c21c0b7
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/registry.rs:502:17
  80:     0x7f5ba464becb - rayon_core::thread_pool::ThreadPool::install::h2cf1be839f111418
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/thread_pool/mod.rs:147:9
  81:     0x7f5ba464becb - polars_mem_engine::executors::projection_utils::window_evaluate::hb5dbd743a674f899
                               at /local/polars/crates/polars-mem-engine/src/executors/projection_utils.rs:54:5
  82:     0x7f5ba464becb - polars_mem_engine::executors::projection_utils::execute_projection_cached_window_fns::{{closure}}::h6250fbb5cc6958fc
                               at /local/polars/crates/polars-mem-engine/src/executors/projection_utils.rs:175:16
  83:     0x7f5ba464becb - rayon_core::join::join::call::{{closure}}::hb87bcdb3b2c779c9
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/join/mod.rs:102:18
  84:     0x7f5ba464becb - rayon_core::join::join_context::call_b::{{closure}}::h925973e0a7d6452c
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/join/mod.rs:129:25
  85:     0x7f5ba464becb - rayon_core::job::StackJob<L,F,R>::run_inline::h4e2f2a3058d4e0f9
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/job.rs:102:9
  86:     0x7f5ba4681bf2 - rayon_core::join::join_context::{{closure}}::h084b999dbf67765e
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/join/mod.rs:159:36
  87:     0x7f5ba468c81f - rayon_core::registry::in_worker::h5dcc60b86af26a51
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/registry.rs:951:13
  88:     0x7f5ba46d3a81 - rayon_core::join::join_context::hf501a90497f6bc5c
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/join/mod.rs:132:5
  89:     0x7f5ba46d3a81 - rayon_core::join::join::h9220fca7d0d43ca8
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/join/mod.rs:105:5
  90:     0x7f5ba468b29a - rayon_core::thread_pool::ThreadPool::join::{{closure}}::hb75065d902e252a2
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/thread_pool/mod.rs:280:25
  91:     0x7f5ba468b29a - rayon_core::thread_pool::ThreadPool::install::{{closure}}::h5164c5388567d265
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/thread_pool/mod.rs:147:40
  92:     0x7f5ba468b29a - rayon_core::registry::Registry::in_worker::hc3232f686c99e7af
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/registry.rs:502:17
  93:     0x7f5ba4672548 - rayon_core::thread_pool::ThreadPool::install::h1e7aa5e727c2aaa5
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/thread_pool/mod.rs:147:9
  94:     0x7f5ba4672548 - rayon_core::thread_pool::ThreadPool::join::hde92aa9c597c79e2
                               at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/thread_pool/mod.rs:280:9
  95:     0x7f5ba4672548 - polars_mem_engine::executors::projection_utils::execute_projection_cached_window_fns::h455b40f57812a0b2
                               at /local/polars/crates/polars-mem-engine/src/executors/projection_utils.rs:173:22
  96:     0x7f5ba4672548 - polars_mem_engine::executors::projection_utils::evaluate_physical_expressions::h2e8d5506040d6d7a
                               at /local/polars/crates/polars-mem-engine/src/executors/projection_utils.rs:236:28
  97:     0x7f5ba46fb8f0 - polars_mem_engine::executors::stack::StackExec::execute_impl::h87098cd1879a4e32
                               at /local/polars/crates/polars-mem-engine/src/executors/stack.rs:46:27
  98:     0x7f5ba46fbe93 - <polars_mem_engine::executors::stack::StackExec as polars_mem_engine::executors::executor::Executor>::execute::h3c0ff3418ab345bd
                               at /local/polars/crates/polars-mem-engine/src/executors/stack.rs:104:13
  99:     0x7f5ba469e2c2 - <polars_mem_engine::executors::slice::SliceExec as polars_mem_engine::executors::executor::Executor>::execute::h5e2d633852570c51
                               at /local/polars/crates/polars-mem-engine/src/executors/slice.rs:17:18
 100:     0x7f5ba43be869 - polars_lazy::frame::LazyFrame::_collect_post_opt::hfc24d90b1b18be8d
                               at /local/polars/crates/polars-lazy/src/frame/mod.rs:678:9
 101:     0x7f5ba43ceb6a - polars_lazy::frame::LazyFrame::collect::hf8b8eef496799001
                               at /local/polars/crates/polars-lazy/src/frame/mod.rs:760:9
 102:     0x7f5ba3e7973b - polars_python::functions::lazy::collect_all::{{closure}}::{{closure}}::{{closure}}::h2abe48d3ca4b0162
                               at /local/polars/crates/polars-python/src/functions/lazy.rs:122:30
 103:     0x7f5ba3e7973b - core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut::hd66594a93b923a1c
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/core/src/ops/function.rs:272:13
 104:     0x7f5ba3e7973b - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once::h18d216e3ba50666b
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/core/src/ops/function.rs:305:13
 105:     0x7f5ba3e7973b - core::option::Option<T>::map::h5809290919daca96
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/core/src/option.rs:1105:29
 106:     0x7f5ba3e7973b - <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next::hacbf2cd2c15e3aaa
                               at /rustc/c6db1ca3c93ad69692a4c4b5542f26fda4bf3aec/library/core/src/iter/adapters/map.rs:107:26
 107:     0x7f5ba3e7973b - <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next::hea936f97cde9029e

Issue description

When upgrading from 1.4.1 -> 1.5.0 (I've also tested on 1.6.0 and main and they both exhibit this behavior). I get this panic when running a moderately complex query. Will try to produce a MRE.

Expected behavior

No panic

Installed versions

``` --------Version info--------- Polars: 1.6.0 Index type: UInt32 Platform: Linux-5.10.224-190.876.x86_64-x86_64-with-glibc2.26 Python: 3.11.7 (main, Dec 5 2023, 22:00:36) [GCC 7.3.1 20180712 (Red Hat 7.3.1-17)] ----Optional dependencies---- adbc_driver_manager altair cloudpickle 3.0.0 connectorx deltalake fastexcel fsspec 2024.6.1 gevent great_tables 0.10.0 matplotlib nest_asyncio 1.6.0 numpy 1.26.4 openpyxl pandas 2.2.2 pyarrow 17.0.0 pydantic 2.9.1 pyiceberg sqlalchemy torch xlsx2csv xlsxwriter ```
EpicUsaMan commented 6 days ago

1) Are you using over statements? Any of your over statements has column with just singular value? 2) Are you using concat somewhere? (lazy after concat can behave like this) 3) Are you using filter with some aggregation functions?

Because I'd same issues in such cases

kszlim commented 6 days ago

1) Yes, I definitely have some window functions 2) Don't believe so 3) Don't believe I have any aggregations

EpicUsaMan commented 6 days ago
  1. Please, check that each value in each over has at least 2 unique values
  2. window functions kinda aggregation as well, I mean something like this pl.col("").filter(pl.col("") > 0).mean().over(...)
  3. Also, I see that you are using collect_all, try to run that separately, just to check, because in 1.5.0/1.6.0 I've a lot of issue this lazy cache, so it can be one of it (I dropped collect_all from my pipeline)
kszlim commented 6 days ago

1) Not sure about this 2) yeah, I think that's likely happening 3) happens even without a collect_all

EpicUsaMan commented 6 days ago

So probably 1 and 2 can be point on which you can build example

Just check your pipeline/data for this two types of entries and try to reduce it only to them

kszlim commented 6 days ago

@ritchie46 still haven't been able to create a repro, but it seems to manifest (in my very complicated query) when I have more than N columns (as opposed to having any particular column cause the issue). Seems like it's also related to having some struct based columns/expressions in the query plan too (as if i remove all of those, it no longer seems to manifest).

kszlim commented 5 days ago

So when running my queries, I get this weirdness:

ldf.select((cs.categorical() | cs.string())).head().collect() # This works fine
ldf.select(~(cs.categorical() | cs.string())).head().collect() # This works fine
ldf.select(cs.string()).head().collect() # This works fine
ldf.select(cs.float()).head().collect() # This works fine
ldf.select((cs.string() | cs.float())).head().collect() # This fails

I'll try to bisect the offending commit between 1.5.0 and 1.4.1. Sounds like some sort of overflow issue.

kszlim commented 5 days ago

Did a git bisect, and it looks like the regression occurs here: https://github.com/pola-rs/polars/pull/18156