We've cargo-culted around a number of configuration options that make our EMR spark conf way more complicated than it needs to be. This task is to reduce the json configuration for the batch-process.sh script at deployment/streaming/scripts/emr-configurations/batch-process.json to a minimum set of options required to successfully complete that task.
Once reduced, we can begin to review other configurations for completeness.
We've cargo-culted around a number of configuration options that make our EMR spark conf way more complicated than it needs to be. This task is to reduce the json configuration for the
batch-process.shscript atdeployment/streaming/scripts/emr-configurations/batch-process.jsonto a minimum set of options required to successfully complete that task.Once reduced, we can begin to review other configurations for completeness.
Original discussion: https://github.com/azavea/osmesa/pull/184#discussion_r367622476