josephmachado / beginner_de_project

Beginner data engineering project - batch edition
https://www.startdataengineering.com/post/data-engineering-project-for-beginners-batch-edition/
MIT License
464 stars 136 forks source link

AWS EMR IAM role error [sent via email] #28

Open josephmachado opened 1 year ago

josephmachado commented 1 year ago

Describe the bug A clear and concise description of what the bug is.

I am trying to get the infra up from more than 3 days. I am terribly stuck at this place. I am working on the beginner batch DE project. Please help me with the error below.

│ Error: error waiting for EMR Cluster (j-1QCC2RIRPI8ZQ) to create: unexpected state 'TERMINATING', wanted target 'RUNNING, WAITING'. last error: VALIDATION_ERROR: EMR service role arn:aws:iam::460318025676:role/EMR_DefaultRole is invalid │ │ with aws_emr_cluster.sde_emr_cluster, │ on main.tf line 108, in resource "aws_emr_cluster" "sde_emr_cluster": │ 108: resource "aws_emr_cluster" "sde_emr_cluster" { │ ╵ ╷ │ Error: could not start transaction: dial tcp 54.90.81.41:5439: connect: connection timed out │ │ with redshift_schema.external_from_glue_data_catalog, │ on main.tf line 170, in resource "redshift_schema" "external_from_glue_data_catalog": │ 170: resource "redshift_schema" "external_from_glue_data_catalog" {


Set up EMR

resource "aws_emr_cluster" "sde_emr_cluster" { name = "sde_emr_cluster" release_label = "emr-6.10.0" applications = ["Spark", "Hadoop"] scale_down_behavior = "TERMINATE_AT_TASK_COMPLETION" service_role = "EMR_DefaultRole" termination_protection = false auto_termination_policy { idle_timeout = var.auto_termination_timeoff }

ec2_attributes { instance_profile = aws_iam_instance_profile.sde_ec2_iam_role_instance_profile.id }

master_instance_group { instance_type = var.instance_type instance_count = 1 name = "Master - 1"

ebs_config {
  size                 = 32
  type                 = "gp2"
  volumes_per_instance = 2
}

}

To Reproduce Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Smartphone (please complete the following information):

Additional context Add any other context about the problem here.