Open ggevay opened 1 year ago
This is unfortunately "works as designed." We don't have any alternative way to handle overflows in the diff. Also, this error should disappear in release mode and simply wrap-around.
Also, this error should disappear in release mode and simply wrap-around.
Then the query could give wrong results, right? (Maybe not this particular one, but some similar query.) Is that ok?
Another repro for this issue is reported here: https://github.com/MaterializeInc/materialize/issues/18645#issuecomment-1722977609
And another repro is in https://github.com/MaterializeInc/materialize/issues/22234.
Also, this error should disappear in release mode and simply wrap-around.
With ASan and release mode this panic is also occurring: https://buildkite.com/materialize/tests/builds/74647#018d6983-9098-4560-9dc2-583a5f307716
thread 'timely:work-1' panicked at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/ops/arith.rs:760:1:
attempt to add with overflow
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::panicking::panic
3: <mz_compute::render::reduce::Accum as differential_dataflow::difference::Semigroup>::plus_equals
4: <alloc::vec::Vec<mz_compute::render::reduce::Accum> as differential_dataflow::difference::Semigroup>::plus_equals
5: differential_dataflow::consolidation::consolidate_updates_from::<(mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>
6: <differential_dataflow::trace::implementations::merge_batcher_col::MergeSorterColumnation<(mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::finish_into
7: <differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::trace::Batcher>::seal::<differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>
8: <timely::dataflow::operators::generic::builder_raw::OperatorCore<mz_repr::timestamp::Timestamp, <timely::dataflow::operators::generic::builder_rc::OperatorBuilder<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>>::build_reschedule<<timely::dataflow::operators::generic::builder_rc::OperatorBuilder<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>>::build<<timely::dataflow::stream::StreamCore<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>> as timely::dataflow::operators::generic::operator::Operator<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>::unary_frontier<alloc::vec::Vec<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}::{closure#0}, timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>>::{closure#0}, <timely::dataflow::stream::StreamCore<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>> as timely::dataflow::operators::generic::operator::Operator<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>::unary_frontier<alloc::vec::Vec<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}::{closure#0}, timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>>::{closure#0}::{closure#0}>::{closure#0}, <timely::dataflow::operators::generic::builder_rc::OperatorBuilder<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>>::build<<timely::dataflow::stream::StreamCore<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>> as timely::dataflow::operators::generic::operator::Operator<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>::unary_frontier<alloc::vec::Vec<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}::{closure#0}, timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>>::{closure#0}, <timely::dataflow::stream::StreamCore<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>> as timely::dataflow::operators::generic::operator::Operator<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>::unary_frontier<alloc::vec::Vec<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}::{closure#0}, timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>>::{closure#0}::{closure#0}>::{closure#0}::{closure#0}>::{closure#0}> as timely::scheduling::Schedule>::schedule
9: <timely::progress::subgraph::PerOperatorState<mz_repr::timestamp::Timestamp>>::schedule
10: <timely::progress::subgraph::Subgraph<mz_repr::timestamp::Timestamp, mz_repr::timestamp::Timestamp> as timely::scheduling::Schedule>::schedule
11: <timely::progress::subgraph::PerOperatorState<mz_repr::timestamp::Timestamp>>::schedule
12: <timely::progress::subgraph::Subgraph<mz_repr::timestamp::Timestamp, mz_repr::timestamp::Timestamp> as timely::scheduling::Schedule>::schedule
13: <timely::progress::subgraph::PerOperatorState<mz_repr::timestamp::Timestamp>>::schedule
14: <timely::progress::subgraph::Subgraph<(), mz_repr::timestamp::Timestamp> as timely::scheduling::Schedule>::schedule
15: <timely::worker::Wrapper>::step
16: <timely::worker::Worker<timely_communication::allocator::generic::Generic>>::step_or_park
17: <mz_compute::server::Worker<timely_communication::allocator::generic::Generic>>::run_client
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Here is another instance in mz_join_core
:
thread 'timely:work-1' panicked at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/ops/arith.rs:346:1:
attempt to multiply with overflow
stack backtrace:
0: rust_begin_unwind
at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/panicking.rs:645:5
1: core::panicking::panic_fmt
at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/panicking.rs:72:14
2: core::panicking::panic
at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/panicking.rs:127:5
3: <i64 as core::ops::arith::Mul>::mul
at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/ops/arith.rs:346:1
4: <i64 as core::ops::arith::Mul<&i64>>::mul
at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/internal_macros.rs:45:17
5: <i64 as differential_dataflow::difference::Multiply>::multiply
at ./cargo/git/checkouts/differential-dataflow-fbfe6a4d66ec22fc/15b7b8d/src/difference.rs:77:53
6: <mz_compute::render::join::mz_join_core::Deferred<timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, differential_dataflow::trace::cursor::cursor_list::CursorList<differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>, differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, mz_repr::row::Row>>::work::<&mut <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, mz_compute::render::join::mz_join_core::mz_join_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, <mz_compute::render::join::linear_join::LinearJoinSpec>::render<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>>::{closure#0}>::{closure#0}::{closure#2}::{closure#5}>::{closure#1}::{closure#0}
at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0a64d05c687e98f0e-1/materialize/tests/src/compute/src/render/join/mz_join_core.rs:633:48
7: <differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>> as differential_dataflow::trace::cursor::Cursor>::map_times::<<mz_compute::render::join::mz_join_core::Deferred<timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, differential_dataflow::trace::cursor::cursor_list::CursorList<differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>, differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, mz_repr::row::Row>>::work<&mut <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, mz_compute::render::join::mz_join_core::mz_join_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, <mz_compute::render::join::linear_join::LinearJoinSpec>::render<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>>::{closure#0}>::{closure#0}::{closure#2}::{closure#5}>::{closure#1}::{closure#0}>
at ./cargo/git/checkouts/differential-dataflow-fbfe6a4d66ec22fc/15b7b8d/src/trace/implementations/ord_neu.rs:462:17
8: <differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>> as differential_dataflow::trace::cursor::Cursor>::map_times::<<mz_compute::render::join::mz_join_core::Deferred<timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, differential_dataflow::trace::cursor::cursor_list::CursorList<differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>, differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, mz_repr::row::Row>>::work<&mut <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, mz_compute::render::join::mz_join_core::mz_join_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, <mz_compute::render::join::linear_join::LinearJoinSpec>::render<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>>::{closure#0}>::{closure#0}::{closure#2}::{closure#5}>::{closure#1}::{closure#0}>
at ./cargo/git/checkouts/differential-dataflow-fbfe6a4d66ec22fc/15b7b8d/src/trace/mod.rs:438:13
9: <mz_compute::render::join::mz_join_core::Deferred<timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, differential_dataflow::trace::cursor::cursor_list::CursorList<differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>, differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, mz_repr::row::Row>>::work::<&mut <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, mz_compute::render::join::mz_join_core::mz_join_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, <mz_compute::render::join::linear_join::LinearJoinSpec>::render<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>>::{closure#0}>::{closure#0}::{closure#2}::{closure#5}>::{closure#1}
at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0a64d05c687e98f0e-1/materialize/tests/src/compute/src/render/join/mz_join_core.rs:631:33
Another one in https://buildkite.com/materialize/tests/builds/77062#018df3dc-85d8-46cf-b5f7-aab307423506
thread 'timely:work-1' panicked at src/compute/src/render/reduce.rs:2018:17:
attempt to add with overflow
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::panicking::panic
3: <mz_compute::render::reduce::Accum as differential_dataflow::difference::Semigroup>::plus_equals
4: differential_dataflow::difference::vector::<impl differential_dataflow::difference::Semigroup for alloc::vec::Vec<R>>::plus_equals
5: differential_dataflow::consolidation::consolidate_updates_from
6: differential_dataflow::trace::implementations::merge_batcher_col::MergeSorterColumnation<D,T,R>::finish_into
7: <differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<K,V,T,D> as differential_dataflow::trace::Batcher>::seal
8: <timely::dataflow::operators::generic::builder_raw::OperatorCore<T,L> as timely::scheduling::Schedule>::schedule
9: timely::progress::subgraph::PerOperatorState<T>::schedule
10: <timely::progress::subgraph::Subgraph<TOuter,TInner> as timely::scheduling::Schedule>::schedule
11: timely::progress::subgraph::PerOperatorState<T>::schedule
12: <timely::progress::subgraph::Subgraph<TOuter,TInner> as timely::scheduling::Schedule>::schedule
13: timely::progress::subgraph::PerOperatorState<T>::schedule
14: <timely::progress::subgraph::Subgraph<TOuter,TInner> as timely::scheduling::Schedule>::schedule
15: timely::worker::Wrapper::step
16: timely::worker::Worker<A>::step_or_park
17: mz_compute::server::Worker<A>::run_client
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Sentry Issue: DATABASE-BACKEND-3BF
Linking this issue, because it contains the same error as #23769, which is marked as duplicate it. This occurred in a production env.
What version of Materialize are you using?
ac9a95501d1e9cc961174ac9a445d7d967840c67
How did you install Materialize?
Built from source
What is the issue?
The i64 Diffs quickly overflow in the following query, causing a crash:
It would be better to report a query-level error.
I guess this doesn't happen so often outside of WMR, but it can happen easily if one uses UNION ALL instead of UNION by mistake in a non-linear recursion, causing an exponential increase in the diffs.
Relevant log output