feat: speed up queries by reading minimal data from homogenous-schema

voltrondata / spark-substrait-gateway

Implements a gateway that speaks the SparkConnect protocol and drives a backend using Substrait (over ADBC Flight SQL).

Apache License 2.0

15 stars 8 forks source link

Closed pthatte1-bb closed 1 month ago

pthatte1-bb commented 1 month ago

Queries on our parquet-datasets (> 1Gb) show < 3 seconds added to every query-execution.

This PR speeds up the execution times.

github-actions[bot] commented 1 month ago

ACTION NEEDED

Substrait follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.