ibis-project / ibis

the portable Python dataframe library
https://ibis-project.org
Apache License 2.0
5.02k stars 585 forks source link

feat: add support for polars, arrow RBR as RHS of a join #9571

Open gforsyth opened 1 month ago

gforsyth commented 1 month ago

We currently support transparent memtable creation from pandas DataFrames and pyarrow Tables if they are provided as the RHS of a join expression. We should extend that to include other data inputs that are supported by memtable, namely polars DataFrames and arrow RecordBatchReaders

jcrist commented 1 month ago

I'd vote to drop support for this instead. For backends that don't have efficient memtables, implicitly creating a memtable multiple times will result in lower performance than calling ibis.memtable once and reusing it. Forcing users to be explicit when coercing other inputs to ibis feels more-correct to me. It's also a bit weird to do this in join methods but not in other table-taking methods like ibis.union.