search

NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
783 stars 228 forks source link

[FEA] create AWS EMR 3.0.1 shim #905

Closed tgravescs closed 3 years ago

tgravescs commented 3 years ago

Is your feature request related to a problem? Please describe. Add a shim layer for aws EMR being based on spark 3.0.1

aluroid commented 3 years ago

It's great to know spark-rapids is going to support EMR, is there an ETA of 0.3.0? Also, latest EMR 6.1.0 is with Spark 3.0.0, so is this feature targeting a future release of EMR?

sameerz commented 3 years ago

We are working to see if we can get the 0.2.0 plugin on EMR 6.2.0, which should be available in November / December.
The target for the 0.3.0 plugin is end of the year.

Data-drone commented 3 years ago

Cloudera Spark 3 release also runs on 3.0.1 so currently I run into: https://docs.cloudera.com/cdp-private-cloud-base/7.1.3/cds-3/topics/spark-spark-3-requirements.html

20/10/14 01:13:34 ERROR repl.Main: Failed to initialize Spark session.
java.lang.IllegalArgumentException: Could not find Spark Shim Loader for 3.0.1.3.0.7110.0-81
    at com.nvidia.spark.rapids.ShimLoader$.detectShimProvider(ShimLoader.scala:45)
    at com.nvidia.spark.rapids.ShimLoader$.findShimProvider(ShimLoader.scala:52)
    at com.nvidia.spark.rapids.ShimLoader$.getSparkShims(ShimLoader.scala:64)
tgravescs commented 3 years ago

thanks for reporting this, we actually have a separate issue to make the version parsing better - https://github.com/NVIDIA/spark-rapids/issues/409

We did add a config that lets the user override the shim if you know it will work. Its marked as an internal config because the user should know what they are doing because if the shim they specify doesn't work things will likely fail. It depends on if each vendor made internal Spark changes. For example you could try: spark.rapids.shims-provider-override=com.nvidia.spark.rapids.shims.spark301.SparkShimServiceProvider

Data-drone commented 3 years ago

cheers that got it to boot but as you said there will be no guarantees that something won't break somewhere. How can I validate?

sameerz commented 3 years ago

You can run the integration tests in your environment (instructions here: https://github.com/NVIDIA/spark-rapids/tree/branch-0.3/integration_tests ). We are going to work with Cloudera to ensure we are compatible in their environment.

Data-drone commented 3 years ago

You can run the integration tests in your environment (instructions here: https://github.com/NVIDIA/spark-rapids/tree/branch-0.3/integration_tests ). We are going to work with Cloudera to ensure we are compatible in their environment.

Okies I'm with Cloudera if you need any help finding people

sameerz commented 3 years ago

I confirmed with our Cloudera contact that CDS 3.0 is basically running Spark 3.0.1, so our plugin will work fine. The workaround mentioned above will work for now. Longer term we will build a shim layer for Cloudera so the workaround is not required.

tgravescs commented 3 years ago

filed https://github.com/NVIDIA/spark-rapids/issues/951 to track that