search

aws / sagemaker-spark-container

The SageMaker Spark Container is a Docker image used to run data processing workloads with the Spark framework on Amazon SageMaker.
Apache License 2.0
36 stars 74 forks source link

fix `test_get_yarn_spark_resource_config` after `AWS_REGION` was added into properties #77

Open pavel-balas opened 2 years ago

pavel-balas commented 2 years ago

Issue #, if available:

The pull/75 change introduced AWS_REGION variables, but they are not reflected in unit test

% pipenv run python -m pytest -s -vv test/unit
...
E         Left contains 2 more items:
E         {'spark.executorEnv.AWS_REGION': 'None',
E          'spark.yarn.appMasterEnv.AWS_REGION': 'None'}
...
FAILED test/unit/test_bootstrapper.py::test_get_yarn_spark_resource_config - assert {'spark.default.parallelism': '8',\n 'spark.driver.defaultJavaOptions': "-XX:OnOutOfMemoryError='kill -9 %p' "\n          ...

Description of changes:

I don't like just update assertion to have string "None", so I'm patching the os variable to have a reasonable value and then asserting that the value made it to the spark properties.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.