NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
822 stars 235 forks source link

Skip `from_json` overflow tests for [databricks] 14.3 #11719

Closed mythrocks closed 1 day ago

mythrocks commented 1 week ago

Fixes #11533.

This commit addresses the test failures reported in #11533, for the following tests:

These failures are a result of #11711. When the JSON parser attempts to read integral struct members from a JSON file, if the parsing leads to an overflow, then the STRUCT column value is deemed null on Databricks 14.3 (i.e. without spark-rapids active). This behaviour differs from that exhibited by Apache Spark versions exceeding 3.4.1.

This commit breaks out the problematic JSON test rows into a separate file, whose read is tested in an xfail for Databricks 14.3. The remaining rows are tested on all versions.

The true fix for #11711 will be addressed later.

mythrocks commented 1 week ago

Build

mythrocks commented 1 week ago

Build

mythrocks commented 1 week ago

Build

mythrocks commented 6 days ago

The CI runs are fine. The failure in (400, 17) is on account of https://github.com/NVIDIA/spark-rapids/issues/11722.