NoSuchMethodError in spark 3.2.4

apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.

https://gluten.apache.org/

Apache License 2.0

1.13k stars 409 forks source link

NoSuchMethodError in spark 3.2.4 #4514

Open weiting-chen opened 7 months ago

weiting-chen commented 7 months ago

Backend

VL (Velox)

Bug description

Run pass with Spark 3.2.4 and it reported NoSuchMethodError.

Spark version

Spark-3.2.x

Spark configurations

Below screenshot is the error message when running spark session:

System information

No response

Relevant logs

No response

PHILO-HE commented 7 months ago

This is a known issue when parquet write is used, as we imported some spark class from 3.2.2/3.3.1 into Gluten for overwriting certain spark logic. Let's clarify it in doc.