[VL] fallback unsupported orc write for spark32 and spark33

apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.

https://gluten.apache.org/

Apache License 2.0

1.22k stars 437 forks source link

[VL] fallback unsupported orc write for spark32 and spark33 #7996

Closed jackylee-ch closed 1 day ago

jackylee-ch commented 2 days ago

I have run bellow sqls failed with footer parse while reading orc file. After look inside, I found the file is written with parquet, not orc, and we haven't support the orc write now.

CREATE TABLE t (c int) USING ORC;
INSERT OVERWRITE TABLE t SELECT 1 as c;
select * from t;

github-actions[bot] commented 2 days ago

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Other pull requests

jackylee-ch commented 1 day ago

cc @PHILO-HE

PHILO-HE commented 1 day ago

I'm not sure about this. Could you add some description? It has no impact on other Spark versions?

jackylee-ch commented 1 day ago

I'm not sure about this. Could you add some description?

Sure, I would update the pr description.

It has no impact on other Spark versions?

For spark32 and spark33, we use GlutenWriterColumnarRules to check the supported fileformats. But for spark34 and spark35, we use BackendsApiManager.getSettings.supportWriteFilesExec to check the supported file formats. Hense this won't impact other spark versions.