Closed NvTimLiu closed 4 days ago
Skip CI, pre-merge job can not cover this change.
Verified the change locally, it works for diff Databricks runtimes.
DB_RUNTIME=14.3
SPARK_VERSION_STR=350
if [[ "$DB_RUNTIME" == "14.3"* ]]; then
DB_SHIM_NAME="${SPARK_VERSION_STR}db143";
else DB_SHIM_NAME="${SPARK_VERSION_STR}db";
fi
echo "Databricks shim name: $DB_SHIM_NAME"
>>Databricks shim name: spark350db143
DB_RUNTIME=13.3
SPARK_VERSION_STR=341
if [[ "$DB_RUNTIME" == "14.3"* ]]; then
DB_SHIM_NAME="${SPARK_VERSION_STR}db143";
else DB_SHIM_NAME="${SPARK_VERSION_STR}db";
fi
echo "Databricks shim name: $DB_SHIM_NAME"
>>Databricks shim name: spark341db
build
To fix: https://github.com/NVIDIA/spark-rapids/issues/11726
Append "143" to the DB14.3 shim name because both Databricks 14.3.x and 15.4.x are based on Spark version 3.5.0