NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
805 stars 234 forks source link

[TASK] cudf dropped python 3.9 support #11394

Closed pxLi closed 1 month ago

pxLi commented 2 months ago

Describe the bug as https://docs.rapids.ai/notices rapids is going to drop python 3.9 support, so all of our related usage require an update

https://github.com/NVIDIA/spark-rapids/blob/branch-24.10/jenkins/Dockerfile-blossom.integration.rocky#L54 https://github.com/NVIDIA/spark-rapids/blob/branch-24.10/jenkins/Dockerfile-blossom.integration.ubuntu#L66

and internal cudf_udf pipelines (databricks + standalone).

lets try make all the defaults to 3.10 for non-CSP environments, and for databricks test cudf_udf only when the default python is supported by rapids

NOTE: if the original environment does not rely on cudf-py (conda,pip), we should keep the python version as pyspark should support all 3.8 and above

Steps/Code to reproduce bug Please provide a list of steps or a code sample to reproduce the issue. Avoid posting private or sensitive data.

Expected behavior A clear and concise description of what you expected to happen.

Environment details (please complete the following information)

Additional context Add any other context about the problem here.

pxLi commented 2 months ago

keep this open to verify cudf_udf cases on other databricks runtimes