aws / sagemaker-spark-container

The SageMaker Spark Container is a Docker image used to run data processing workloads with the Spark framework on Amazon SageMaker.
Apache License 2.0
36 stars 74 forks source link

feature: upgrade to emr 6.1.0 #45

Closed guoqiao1992 closed 3 years ago

guoqiao1992 commented 3 years ago

upgrade to emr 6.1.0(spark 3.0.0) to support hadoop 3.2.1

Description of changes:

use different partition for PDT use al2 from ecr extend timeout in job generalize copy_aws_jar and upgrade emr 6.1.0

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.