feat: expression level definition of CSV and Parquet data sources

[kszucs]

Is your feature request related to a problem?

I'd like to be able to build up an expression based on local files and easily execute it on multiple backends:

t = ibis.read_parquet("...path to parquet file...", name="t")

expr = ...  # build up some expression depending on the source parquet file

# execute it using DuckDB
ibis.duckdb.execute(expr)

# decide to use a different backend later on
ibis.polars.execute(expr)

Describe the solution you'd like

The expression system should contain specific operations describing various data sources. This way ibis expression can be built in a backend agnostic manner.

What version of ibis are you running?

main branch

What backend(s) are you using, if any?

DuckDB, Polars, DataFusion, Pandas

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[lostmygithubaccount]

what's your use case for this out of curiosity?

[cpcloud]

The primary technical challenge here is finding a fast and robust schema inference solution for files that aren't self-describing such as CSV and JSON.

The current landscape is a bit dicey:

CSV

• Can be done with pyarrow or DuckDB, unclear what the differences are

JSON

• Schema inference without reading the entire file can only be done with DuckDB AFAICT, pyarrow doesn't have a solution for this, you have to read everything in with pyarrow.json.read_json which defeats the purpose of modeling these operations as expressions.