Update the aws_emr_instance_group resource to support non TASK instance groups

[nywilken]

The aws_emr_instance_group resource previously known as the aws_emr_task_group has diverged since introducing the idea of instance groups to the aws_emr_cluster resource. Which has lead to confusion, and update issues, when trying to manage instance groups from within a provisioned aws_emr_cluster.

This request is to update aws_emr_instance_gruop to have parity with the instance group configuration capabilities in aws_emr_cluster so that Terraform users can begin managing instance groups separately from the aws_emr_cluster resource.

A sample configuration would look something like the following

resource "aws_emr_cluster" "test" {                                                                                                                                                                                                                                                                                                                                                           
  name             = "tf-emr-test"                                                                                                                                                                                                                                                                                                                                                        
  release_label    = "emr-5.14.0"                                                                                                                                                                                                                                                                                                                                                             
  applications     = ["spark", "hadoop", "sqoop"]                                                                                                                                                                                                                                                                                                                                             
  service_role     = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/EMR_DefaultRole"                                                                                                                                                                                                                                                                                       
  autoscaling_role = "EMR_AutoScaling_DefaultRole"                                                                                                                                                                                                                                                                                                                                            

  ec2_attributes {                                                                                                                                                                                                                                                                                                                                                                            
    subnet_id        = "${aws_subnet.main.id}"                                                                                                                                                                                                                                                                                                                                                
    instance_profile = "${aws_iam_instance_profile.test_profile.arn}"                                                                                                                                                                                                                                                                                                                         
  }
}

resource "aws_emr_instance_group" "master" {
   cluster_id     = "${aws_emr_cluster.test.id}"                                                                                                                                                                                                                                                                                                                                                                            
    instance_role  = "MASTER"                                                                                                                                                                                                                                                                                                                                                                 
    instance_type  = "m4.large"                                                                                                                                                                                                                                                                                                                                                               
    instance_count = 1                                                                                                                                                                                                                                                                                                                                                                        

    ebs_config {                                                                                                                                                                                                                                                                                                                                                                              
      size                 = "25"                                                                                                                                                                                                                                                                                                                                                             
      type                 = "gp2"                                                                                                                                                                                                                                                                                                                                                            
      volumes_per_instance = 1                                                                                                                                                                                                                                                                                                                                                                
    }                                                                                                                                                                                                                                                                                                                                                                                         
  } 
}

resource "aws_emr_instance_group" "core" {
   cluster_id     = "${aws_emr_cluster.test.id}"                                                                                                                                                                                                                                                                                                                                                                            
    instance_role  = "CORE"                                                                                                                                                                                                                                                                                                                                                                 
    instance_type  = "m4.large"                                                                                                                                                                                                                                                                                                                                                               
    instance_count = 1                                                                                                                                                                                                                                                                                                                                                                        

    ebs_config {                                                                                                                                                                                                                                                                                                                                                                              
      size                 = "25"                                                                                                                                                                                                                                                                                                                                                             
      type                 = "gp2"                                                                                                                                                                                                                                                                                                                                                            
      volumes_per_instance = 1                                                                                                                                                                                                                                                                                                                                                                
    }                                                                                                                                                                                                                                                                                                                                                                                         
  } 
}

resource "aws_emr_instance_group" "task" {
   cluster_id     = "${aws_emr_cluster.test.id}"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
    instance_role  = "TASK"                                                                                                                                                                                                                                                                                                                                                                   
    instance_type  = "m4.large"                                                                                                                                                                                                                                                                                                                                                               
    instance_count = 2                                                                                                                                                                                                                                                                                                                                                                        

    autoscaling_policy = <<EOF
{
  "Constraints": {
    "MinCapacity": 1,
    "MaxCapacity": 2
  },
  "Rules": [
    {
      "Name": "ScaleOutMemoryPercentage",
      "Description": "Scale out if YARNMemoryAvailablePercentage is less than 15",
      "Action": {
        "SimpleScalingPolicyConfiguration": {
          "AdjustmentType": "CHANGE_IN_CAPACITY",
          "ScalingAdjustment": 1,
          "CoolDown": 300
        }
      },
      "Trigger": {
        "CloudWatchAlarmDefinition": {
          "ComparisonOperator": "LESS_THAN",
          "EvaluationPeriods": 1,
          "MetricName": "YARNMemoryAvailablePercentage",
          "Namespace": "AWS/ElasticMapReduce",
          "Period": 300,
          "Statistic": "AVERAGE",
          "Threshold": 15,
          "Unit": "PERCENT"
        }
      }
    }
  ]
} 
EOF                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                                                                                                                         }

[omerfsen]

How can we tag aws_emr_instance_group as before it was under aws_emr_cluster and we can use tag but now when i look to https://www.terraform.io/docs/providers/aws/r/emr_instance_group.html I don't see we can use "tag" Argument

[stereoj]

is any way to tag aws_emr_instance_group as @omerfsen asked before? there is no tag information in documentation: https://www.terraform.io/docs/providers/aws/r/emr_instance_group.html @omerfsen did you find any soultion or workaround how to do this? I'll appreciate any help and info.

[bflad]

Hi folks 👋 Due to limitations in the EMR API, we are going to stick with the current implementation of the master_instance_group and core_instance_group configuration blocks built into the aws_emr_cluster resource. If you have other feature requests or bug reports with EMR functionality, please raise a new GitHub issue and we will take a fresh look. Thanks.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!