search

NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
792 stars 230 forks source link

Fix tests failures in join_test.py #11022

Open razajafri opened 3 months ago

razajafri commented 3 months ago 
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_broadcast_hash_join_fix_fallback_by_inputfile
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_broadcast_nested_join_fix_fallback_by_inputfile
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_broadcast_nested_loop_join_special_case_count
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_broadcast_nested_loop_join_special_case_group_by_count
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_broadcast_nested_loop_join_with_condition
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_cartesian_join_special_case_count
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_cartesian_join_special_case_group_by_count
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_distinct_join
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_half_cache_join
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_left_broadcast_nested_loop_join_condition_missing_count
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_right_broadcast_nested_loop_join_condition_missing_count
mythrocks commented 3 months ago

Two of these tests (test_broadcast_hash_join_fix_fallback_by_inputfile, and test_broadcast_nested_join_fix_fallback_by_inputfile) seem to be failing with the same missing gpuDecodeTime metric:

E                   Caused by: java.util.NoSuchElementException: key not found: gpuDecodeTime
E                       at scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:243)
E                       at scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:239)
E                       at com.nvidia.spark.rapids.OrcTableReader.next(GpuOrcScan.scala:2884)
E                       at com.nvidia.spark.rapids.OrcTableReader.next(GpuOrcScan.scala:2859)

This comes from the ORC file reader.

The rest of the tests seem to run fine, when ANSI mode is disabled.

mythrocks commented 2 months ago

I've unassigned myself from this bug. This should be resolved after the missing metrics are addressed.