search

apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
https://gluten.apache.org/
Apache License 2.0
1.16k stars 420 forks source link

Velox doesn't work with Spark Delta Lake #2891

Closed felipepessoto closed 10 months ago

felipepessoto commented 1 year ago

Backend

VL (Velox)

Bug description

When I read a Delta Lake table it fallback:

GlutenFallbackReporter [Thread-32]: Validation failed for plan: Scan parquet , due to: does not support fileFormat: UnknownFormat.

Delta Lake tables are built on top of Parquet

Spark version

Spark-3.3.x

Spark configurations

No response

System information

No response

Relevant logs

No response

felipepessoto commented 9 months ago

@YannByron, have you tried Delta Lake 2.4? Without DV, which is not supported by Gluten yet.