search

Yelp / mrjob

Run MapReduce jobs on Hadoop or Amazon Web Services
http://packages.python.org/mrjob/
Other
2.61k stars 587 forks source link

extra_cluster_params should merge dictionaries #2154

Closed coyotemarin closed 4 years ago

coyotemarin commented 4 years ago

Currently, if you do something like this in your config:

runners:
  emr:
    extra_cluster_params:
      Instances:
        EmrManagedMasterSecurityGroup: sg-00000000000000000

it'll actually blow away the entire Instances parameter to RunJobFlow, rather than just setting the security group. This is almost certainly not what users expect.