Open ava6969 opened 1 year ago
There is a hash-join operation in Acero if you want to use that (note, this example uses some convenience utilities that were only added in 11.0.0, the operation is possible in 10.0.0 or earlier but more complex):
cp::Declaration left{"table_source", cp::TableSourceNodeOptions{left_table}};
cp::Declaration right{"table_source", cp::TableSourceNodeOptions{right_table}};
cp::HashJoinNodeOptions join_opts{
cp::JoinType::INNER,
/*left_keys=*/{"index"},
/*right_keys=*/{"index"}, cp::literal(true), "l_", "r_"};
cp::Declaration hashjoin{
"hashjoin", {std::move(left), std::move(right)}, std::move(join_opts)};
ARROW_ASSIGN_OR_RAISE(std::shared_ptr<Table> joined_table, cp::DeclarationToTable(std::move(hashjoin));
A full working example is in https://github.com/apache/arrow/blob/master/cpp/examples/arrow/execution_plan_documentation_examples.cc
Describe the usage question you have. Please include as many useful details as possible.
So I implemented a very inefficient operation to con-cat 2 or more record batches that all individually have an array holding their index. I have been reading about hash operations. I need help improving my current implementation and using arrow functions.
This is fast as expected but it loses information about the index of each batch
Component(s)
C++