spiraldb / vortex

A toolkit for working with compressed Arrow in-memory, on-disk, and over-the-wire
Apache License 2.0
92 stars 5 forks source link

[WIP] Datafusion pushdown support #413

Closed a10y closed 1 week ago

a10y commented 1 week ago

Unfinished, just opening this as I continue to get things working.

This PR augments the original Vortex connection for Datafusion, with an implementation of filter pushdown that allows us to perform late materialization on as many columns as possible.

Pushdown support will be able to get flagged on/off so we can run benchmarks testing different strategies.

I'm hoping to have an initial version of this with a benchmark harness tonight.