[BUG] Unit test failures in Spark 3.2 shim build

[gerashegalov]

Describe the bug Umbrella issue to fix unit tests in the 3.2 shim

Steps/Code to reproduce bug

$ mvn clean package -Dbuildver=320 -Dskip  

Expected behavior no unit test failures

Environment details (please complete the following information) local

Additional context

3323

Tasks:

HashAggregatesSuite

Fixed by https://github.com/NVIDIA/spark-rapids/pull/3403

• [x] WITH DECIMALS: Avg Distinct with filter FAILED

CastOpSuite

See https://github.com/NVIDIA/spark-rapids/issues/3382 for more detail.

Fixed by https://github.com/NVIDIA/spark-rapids/pull/3439 and https://github.com/NVIDIA/spark-rapids/pull/3531

• [x] Cast from string to date using random inputs FAILED
• [x] Cast from string to date using random inputs with valid year prefix FAILED
• [x] Cast from string to timestamp FAILED
• [x] Test all supported casts with in- [ ] range values FAILED

ExpandExecSuite

Fixed by https://github.com/NVIDIA/spark-rapids/pull/3403

• [x] IGNORE ORDER, WITH DECIMALS: group with aggregates FAILED

WindowFunctionSuite

Filed as https://github.com/NVIDIA/spark-rapids/issues/3415

Fixed by https://github.com/NVIDIA/spark-rapids/pull/3547

• [x] WITH DECIMALS: [Window] [RANGE] [ ASC] [- [ ] 2 DAYS, 3 DAYS] FAILED
• [x] WITH DECIMALS: [Window] [RANGE] [DESC] [- [ ] 2 DAYS, 3 DAYS] FAILED
• [x] WITH DECIMALS: [Window] [RANGE] [ ASC] [- [ ] 2 DAYS, CURRENT ROW] FAILED
• [x] WITH DECIMALS: [Window] [RANGE] [DESC] [- [ ] 2 DAYS, CURRENT ROW] FAILED
• [x] WITH DECIMALS: [Window] [RANGE] [ ASC] [CURRENT ROW, 3 DAYS] FAILED
• [x] WITH DECIMALS: [Window] [RANGE] [DESC] [CURRENT ROW, 3 DAYS] FAILED
• [x] IGNORE ORDER, WITH DECIMALS: [Window] [MIXED WINDOW SPECS] FAILED

ParquetWriterSuite

These tests were fixed by https://github.com/NVIDIA/spark-rapids/pull/3377

• [x] Job commit time metrics FAILED

CostBasedOptimizerSuite

These tests were fixed by https://github.com/NVIDIA/spark-rapids/pull/3377 and SPARK-36666

• [x] keep CustomShuffleReaderExec on GPU FAILED
• [x] Compute estimated row count nested joins no broadcast FAILED

These tests need updating and have been ignored for now. There is a follow-on issue https://github.com/NVIDIA/spark-rapids/issues/3526 to update them when we resume work on CBO.

• [x] Avoid transition to GPU for trivial projection after CPU SMJ FAILED
• [x] Force section of plan back onto CPU, AQE on FAILED
• [x] Force last section of plan back onto CPU, AQE on FAILED

AdaptiveQueryExecSuite

These tests were fixed by https://github.com/NVIDIA/spark-rapids/pull/3377 and SPARK-36666

• [x] Plugin should translate child plan of GPU DataWritingCommandExec to GPU FAILED
• [x] Plugin should translate child plan of CPU DataWritingCommandExec to GPU FAILED
• [x] Exchange reuse FAILED
• [x] Change merge join to broadcast join without local shuffle reader FAILED
• [x] Verify the reader is LocalShuffleReaderExec FAILED
• [x] Avoid splitting CustomShuffleReader and ShuffleQueryStageExec pairs FAILED

KnownNotNullSuite

Fixed by https://github.com/NVIDIA/spark-rapids/pull/3457

• [x] KnownNotNull in Scala UDF- [ ] Int FAILED
• [x] KnownNotNull in Scala UDF- [ ] Long FAILED
• [x] KnownNotNull in Scala UDF- [ ] Short FAILED

AnsiCastOpSuite

These tests were fixed by https://github.com/NVIDIA/spark-rapids/pull/3377

• [x] WITH DECIMALS: Write bytes to string FAILED
• [x] WITH DECIMALS: Write shorts to string FAILED
• [x] WITH DECIMALS: Write ints to string FAILED
• [x] WITH DECIMALS: Write longs to string FAILED
• [x] WITH DECIMALS: Write ints to long FAILED
• [x] WITH DECIMALS: Write longs to int (values within range) FAILED
• [x] WITH DECIMALS: Write longs to short (values within range) FAILED
• [x] WITH DECIMALS: Write longs to byte (values within range) FAILED
• [x] WITH DECIMALS: Write ints to short (values within range) FAILED
• [x] WITH DECIMALS: Write ints to byte (values within range) FAILED
• [x] WITH DECIMALS: Write shorts to byte (values within range) FAILED
• [x] WITH DECIMALS: Write floats to long (values within range) FAILED
• [x] WITH DECIMALS: Write floats to int (values within range) FAILED
• [x] WITH DECIMALS: Write floats to short (values within range) FAILED
• [x] WITH DECIMALS: Write floats to byte (values within range) FAILED
• [x] WITH DECIMALS: Write doubles to long (values within range) FAILED
• [x] WITH DECIMALS: Write doubles to int (values within range) FAILED
• [x] WITH DECIMALS: Write doubles to short (values within range) FAILED
• [x] WITH DECIMALS: Write doubles to byte (values within range) FAILED
• [x] WITH DECIMALS: Copy ints to long FAILED
• [x] WITH DECIMALS: Copy long to float FAILED

ParseDateTimeSuite

See https://github.com/NVIDIA/spark-rapids/issues/3383 for more detail. fixed by https://github.com/NVIDIA/spark-rapids/pull/3439

• [x] WITH DECIMALS: to_date yyyy- [ ] MM- [ ] dd FAILED
• [x] WITH DECIMALS: to_date dd/MM/yyyy FAILED
• [x] WITH DECIMALS: to_date default pattern FAILED
• [x] parse now FAILED

ScalarSubquerySuite

Failures due to https://github.com/NVIDIA/spark-rapids/issues/3400 (canonicalizationMatchesCpu=true != canonicalizationMatchesGpu=false)

• [x] WITH DECIMALS: Uncorrelated Scalar Subquery FAILED

[andygrove]

I am looking at the AnsiCastOpSuite failures with the pattern WITH DECIMALS: Write _ to _.

[andygrove]

I also started looking at CastOpSuite and filed a separate issue since this looks like it could be quite involved. https://github.com/NVIDIA/spark-rapids/issues/3382

[andygrove]

I filed https://github.com/NVIDIA/spark-rapids/issues/3383 for ParseDateTimeSuite failures

[tgravescs]

cast ops is fixed by 3531 and no other unit tests failures currently

[tgravescs]

3.2 unit tests all pass now