Whitelist AWS_REGION in yarn-site

aws / sagemaker-spark-container

The SageMaker Spark Container is a Docker image used to run data processing workloads with the Spark framework on Amazon SageMaker.

Apache License 2.0

36 stars 74 forks source link

Whitelist AWS_REGION in yarn-site #75

Closed can-sun closed 2 years ago

can-sun commented 2 years ago

Issue #, if available: Fix the issue that AWS_REGION is still not available in worker node when using multi-instances.

Description of changes: Whitelist AWS_REGION in yarn-site.xml, rerun the test and verified processing job with multiple instances can run successfully. And also fixed the python format check failure

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

pavel-balas commented 2 years ago

Hi, shouldn't this change cause test_get_yarn_spark_resource_config test to fail as exp_spark_config_props doesn't have AWS_REGION (code)?