-
It would be neat if we could create Substrait expressions from Ibis expressions. One use case would be to project/filter pyarrow datasets. See https://gist.github.com/ianmcook/f70fc185d29ae97bdf85ffe0…
-
### Is your feature request related to a problem or challenge?
A report from Twitter https://twitter.com/mim_djo/status/1740542585410814393
Says:
> a new release of #datafusion 34, still readi…
alamb updated
4 months ago
-
[Substrait](https://substrait.io/) describes itself as **Substrait: Cross-Language Serialization for Relational Algebra** and its vision is to:
> Create a well-defined, cross-language [specificatio…
-
Usages of the SparkSQL function [`def struct(cols: Column*): Column`](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.struct.html) fail with an error me…
-
substrait doesn't keep track of intermediate field names, which makes it hard to properly round-trip queries. Now there's a way to optionally include names in `output_names` field as a [hint](https://…
-
I would like to use the SetRel operator to union two datasets. I got the error: `NotImplemented: conversion to arrow::compute::Declaration from Substrait relation set{...}`. According to the `relation…
-
This is a general feature to transform a substrait plan into some executable code.
A substrait plan is expected to be a serialized protobuf message.
Desired expression formats include the follow…
-
### Is your feature request related to a problem or challenge?
While working on an example for serializing substrait plans (see https://github.com/apache/arrow-datafusion/pull/9260 PR), I found I c…
alamb updated
5 months ago
-
The existing FlightSQLExecutor executes a sql string. We should add support for executing a serialized Substrait plan. In the longer term , it should be easier to support more advanced pushdown for en…
-
**Is your feature request related to a problem? Please describe.**
substrait-io is a project that aims to offer cross-language serialization for relational algebra. DataFusion has a producer and cons…