Is your feature request related to a problem? Please describe.
currently we do not support inequality joins (hash join and sort-merge join). it is hard to implement such feature because datafusion has no direct supports to row-based evaluation.
Describe the solution you'd likeDescribe alternatives you've considered
simulate row-based evaluation with one-row columnar evaluation, which has super low performance in practice. in cases where the equality pred has filtered away most records, this method may work. but if the equality pred takes no effects (like tpcds q72). the query will hang.
supports limited row-based filter in datafusion. currently datafusion already has some supports like make_comparator to build a row-based evaluator. we can extend it to support more row-based evaluations, like make_binary_op etc.
fallback the post-filter evaluation to spark, and use codegen to speedup the evaluation. but we also have to consider the fallback overheads.
Is your feature request related to a problem? Please describe. currently we do not support inequality joins (hash join and sort-merge join). it is hard to implement such feature because datafusion has no direct supports to row-based evaluation.
Describe the solution you'd like Describe alternatives you've considered
make_comparatorto build a row-based evaluator. we can extend it to support more row-based evaluations, likemake_binary_opetc.Additional context