search

NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
822 stars 235 forks source link

Fix Parquet Writer tests on [databricks] 14.3 #11673

Closed mythrocks closed 2 weeks ago

mythrocks commented 3 weeks ago

Fixes #11534.

This commit fixes the test failures in parquet_write_test.py, as listed on #11534.

This is an extension of the changes made in #11615, which attempted to address similar failures on Apache Spark 4.

Most of the test failures pertain to legacy Parquet writer settings and conf keys which were removed on Spark 4. A stray test involves a change in how the test gleans description strings from exceptions.

mythrocks commented 3 weeks ago

Build

mythrocks commented 3 weeks ago

Build

mythrocks commented 2 weeks ago

Build

mythrocks commented 2 weeks ago

@razajafri, I've moved the tests over to use the new conf names. Good suggestion. This reduces the code on the test-side.

If this is agreeable, shall we check this in? (We'll ignore the failure against Spark 4.0 for the moment.)

mythrocks commented 2 weeks ago

Thank you for the reviews, @revans2, @razajafri. I've merged this change as well.