Adds support for running Spark executors in Docker containers on EMR. It's activated by setting the docker_image option. Also added the docker_mounts option, and the docker_client_config option (so you can log into ECR). Fixes #2179.
If you want to test this out without building a Docker image, try python -m mrjob.examples.mr_spark_wordcount --docker-image amazoncorretto:8 on Python 2.
Adds support for running Spark executors in Docker containers on EMR. It's activated by setting the
docker_image
option. Also added thedocker_mounts
option, and thedocker_client_config
option (so you can log into ECR). Fixes #2179.If you want to test this out without building a Docker image, try
python -m mrjob.examples.mr_spark_wordcount --docker-image amazoncorretto:8
on Python 2.