NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
822 stars 235 forks source link

Change Databricks 14.3 shim name to spark350db143 [skip ci] #11728

Closed NvTimLiu closed 4 days ago

NvTimLiu commented 4 days ago

To fix: https://github.com/NVIDIA/spark-rapids/issues/11726

Append "143" to the DB14.3 shim name because both Databricks 14.3.x and 15.4.x are based on Spark version 3.5.0

NvTimLiu commented 4 days ago

Skip CI, pre-merge job can not cover this change.

Verified the change locally, it works for diff Databricks runtimes.

DB_RUNTIME=14.3
SPARK_VERSION_STR=350
if [[ "$DB_RUNTIME" == "14.3"* ]]; then
     DB_SHIM_NAME="${SPARK_VERSION_STR}db143";
else     DB_SHIM_NAME="${SPARK_VERSION_STR}db";
fi
echo "Databricks shim name: $DB_SHIM_NAME"
>>Databricks shim name: spark350db143

DB_RUNTIME=13.3
SPARK_VERSION_STR=341
if [[ "$DB_RUNTIME" == "14.3"* ]]; then
     DB_SHIM_NAME="${SPARK_VERSION_STR}db143";
else     DB_SHIM_NAME="${SPARK_VERSION_STR}db";
fi
echo "Databricks shim name: $DB_SHIM_NAME"
>>Databricks shim name: spark341db
NvTimLiu commented 4 days ago

build