terraform-aws-modules / terraform-aws-emr

Terraform module to create AWS EMR resources πŸ‡ΊπŸ‡¦
https://registry.terraform.io/modules/terraform-aws-modules/emr/aws
Apache License 2.0
23 stars 19 forks source link

Service Pass Role does not support custom instance profile #26

Open tadeha opened 2 months ago

tadeha commented 2 months ago

Description

When disabling create_iam_instance_profile, the Terraform apply will fail due to aws_iam_policy.service_pass_role not containing any resource.

SCR-20240703-ktdq

Versions

Reproduction Code [Required]

module "emr_test_2" {
  source = "terraform-aws-modules/emr/aws"

  name = "example-instance-fleet"

  version = "~> 2.1.0"

  create_iam_instance_profile = false

  release_label = "emr-6.9.0"
  applications  = ["spark", "trino"]
  auto_termination_policy = {
    idle_timeout = 3600
  }

  bootstrap_action = {
    example = {
      path = "file:/bin/echo",
      name = "Just an example",
      args = ["Hello World!"]
    }
  }

  configurations_json = jsonencode([
    {
      "Classification" : "spark-env",
      "Configurations" : [
        {
          "Classification" : "export",
          "Properties" : {
            "JAVA_HOME" : "/usr/lib/jvm/java-1.8.0"
          }
        }
      ],
      "Properties" : {}
    }
  ])

  master_instance_fleet = {
    name                      = "master-fleet"
    target_on_demand_capacity = 1
    instance_type_configs = [
      {
        instance_type = "m5.xlarge"
      }
    ]
  }

  core_instance_fleet = {
    name                      = "core-fleet"
    target_on_demand_capacity = 2
    target_spot_capacity      = 2
    instance_type_configs = [
      {
        instance_type     = "c4.large"
        weighted_capacity = 1
      },
      {
        bid_price_as_percentage_of_on_demand_price = 100
        ebs_config = [{
          size                 = 256
          type                 = "gp3"
          volumes_per_instance = 1
        }]
        instance_type     = "c5.xlarge"
        weighted_capacity = 2
      },
      {
        bid_price_as_percentage_of_on_demand_price = 100
        instance_type                              = "c6i.xlarge"
        weighted_capacity                          = 2
      }
    ]
    launch_specifications = {
      spot_specification = {
        allocation_strategy      = "capacity-optimized"
        block_duration_minutes   = 0
        timeout_action           = "SWITCH_TO_ON_DEMAND"
        timeout_duration_minutes = 5
      }
    }
  }

  task_instance_fleet = {
    name                      = "task-fleet"
    target_on_demand_capacity = 1
    target_spot_capacity      = 2
    instance_type_configs = [
      {
        instance_type     = "c4.large"
        weighted_capacity = 1
      },
      {
        bid_price_as_percentage_of_on_demand_price = 100
        ebs_config = [{
          size                 = 256
          type                 = "gp3"
          volumes_per_instance = 1
        }]
        instance_type     = "c5.xlarge"
        weighted_capacity = 2
      }
    ]
    launch_specifications = {
      spot_specification = {
        allocation_strategy      = "capacity-optimized"
        block_duration_minutes   = 0
        timeout_action           = "SWITCH_TO_ON_DEMAND"
        timeout_duration_minutes = 5
      }
    }
  }

  ebs_root_volume_size = 64
  ec2_attributes = {
    # Subnets should be private subnets and tagged with
    # { "for-use-with-amazon-emr-managed-policies" = true }
    subnet_ids       = ["subnet-abcde012", "subnet-bcde012a", "subnet-fghi345a"]
    instance_profile = "some_arn"
  }
  vpc_id = "vpc-1234556abcdef"

  list_steps_states = ["PENDING", "RUNNING", "FAILED", "INTERRUPTED"]
  log_uri           = "s3://my-elasticmapreduce-bucket/"

  scale_down_behavior    = "TERMINATE_AT_TASK_COMPLETION"
  step_concurrency_level = 3
  termination_protection = false
  visible_to_all_users   = true

  tags = {
    Terraform   = "true"
    Environment = "dev"
  }
}

Expected behavior

I would suppose, here: https://github.com/terraform-aws-modules/terraform-aws-emr/blob/90689fde717d68d901ad6d7f7f1638d85595bf2b/main.tf#L618 , based on the create_iam_instance_profile flag, it should put the correct instance profile ARN.

Actual behavior

But, currently, it just behaves correctly when we create the instance profile via the module.

Terminal Output Screenshot(s)

image

tadeha commented 1 month ago

Could you please let us know if this project is currently being maintained? @antonbabenko We have been blocked by an issue for weeks. Thank you.

bryantbiggs commented 1 month ago

It is maintained - but we don't work on-demand for free so you'll have to wait until someone has bandwidth to take a look 😬

github-actions[bot] commented 1 week ago

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

tadeha commented 1 week ago

Not stale.