AWS ALB target groups not found during 0.12 upgrade

npc-adrian commented 5 years ago

While upgrading an AWS load Balancer from terraform 0.11 to terraform 0.12, I am getting an error when I try to run terraform plan against 0.12 for the first time.

Error: Error retrieving Target Group: ValidationError: 'arn:aws:elasticloadbalancing:eu-west-1:000000000000:targetgroup/dev-dev2-blog/a3f1958ff217c688' is not a valid target group ARN
    status code: 400, request id: 6f8ab670-910b-11e9-83dd-efb1ea6a619f

Everything appears to be in order - the resources exist, the state files are not corrupted etc. Details below.

I can't tell if this is a problem with the upgrade or the AWS provider. Apologies if I've added it in the wrong place.

Terraform Version

Terraform v0.12.2
+ provider.aws v2.15.0

Terraform Configuration Files

resource "aws_lb_target_group" "arthr" {
  name     = "${var.stage}-${var.environment}-arthr"
  port     = 80
  protocol = "HTTP"
  vpc_id   = var.vpc_id

  health_check {
    healthy_threshold   = 5
    unhealthy_threshold = 2
    path                = "/check"
    matcher             = "200"
  }
}

resource "aws_lb_target_group" "blog" {
  name     = "${var.stage}-${var.environment}-blog"
  port     = 80
  protocol = "HTTP"
  vpc_id   = var.vpc_id
}

Debug Output

https://gist.github.com/npc-adrian/85024e84da8ae2d064245cfa9b8c3386

Crash Output

$ tf12 plan > /dev/null

Error: Error retrieving Target Group: ValidationError: 'arn:aws:elasticloadbalancing:eu-west-1:000000000000:targetgroup/dev-dev2-blog/a3f1958ff217c688' is not a valid target group ARN
    status code: 400, request id: 6f8ab670-910b-11e9-83dd-efb1ea6a619f

Error: Error retrieving Target Group: ValidationError: 'arn:aws:elasticloadbalancing:eu-west-1:000000000000:targetgroup/dev-dev2-arthr/55ccc73b8681221d' is not a valid target group ARN
    status code: 400, request id: 6f96eb40-910b-11e9-a808-67e629eff967

Expected Behavior

The plan should succeed.

Actual Behavior

The plan failed.

Steps to Reproduce

Upgraded to terraform 0.12
Upgraded all modules
Ran terraform plan using 0.12

Additional Context

I followed the instructions to upgrade from 0.11 to 0.12. It has worked for other terraform projects so I'm pretty sure I did everything right.

The plan fails to find 2 AWS Load Balancer target groups but they are definitely correct when I look in the console. Here are the entries from the state file...

$ tf12 state show module.sunset_environment.aws_lb_target_group.blog | grep arn
    arn                  = "arn:aws:elasticloadbalancing:eu-west-1:000000000000:targetgroup/dev-dev2-blog/a3f1958ff217c688"
    arn_suffix           = "targetgroup/dev-dev2-blog/a3f1958ff217c688"
    id                   = "arn:aws:elasticloadbalancing:eu-west-1:000000000000:targetgroup/dev-dev2-blog/a3f1958ff217c688"

and

$ tf12 state show module.sunset_environment.aws_lb_target_group.arthr
    arn                  = "arn:aws:elasticloadbalancing:eu-west-1:000000000000:targetgroup/dev-dev2-arthr/55ccc73b8681221d"
    arn_suffix           = "targetgroup/dev-dev2-arthr/55ccc73b8681221d"
    id                   = "arn:aws:elasticloadbalancing:eu-west-1:000000000000:targetgroup/dev-dev2-arthr/55ccc73b8681221d"

I confirmed I could look them up using the AWS CLI. Here's one example.

$ aws elbv2 describe-target-groups --target-group-arns 'arn:aws:elasticloadbalancing:eu-west-1:000000000000:targetgroup/dev-dev2-blog/a3f1958ff217c688'
  {
      "TargetGroups": [
          {
              "TargetGroupArn": "arn:aws:elasticloadbalancing:eu-west-1:000000000000:targetgroup/dev-dev2-blog/a3f1958ff217c688",
              "TargetGroupName": "dev-dev2-blog",
              "Protocol": "HTTP",
              "Port": 80,
              "VpcId": "vpc-xxxxxxxxxxxxxxxxx",
              "HealthCheckProtocol": "HTTP",
              "HealthCheckPort": "traffic-port",
              "HealthCheckEnabled": true,
              "HealthCheckIntervalSeconds": 30,
              "HealthCheckTimeoutSeconds": 5,
              "HealthyThresholdCount": 5,
              "UnhealthyThresholdCount": 2,
              "HealthCheckPath": "/",
              "Matcher": {
                  "HttpCode": "200"
              },
              "LoadBalancerArns": [
                  "arn:aws:elasticloadbalancing:eu-west-1:000000000000:loadbalancer/app/dev-dev2/9e3252cff6a19475"
              ],
              "TargetType": "instance"
          }
      ]
  }

I also checked that when I can re-initialize and run terraform plan against 0.11 sucessfully and that it does not find any changes to be made. Here are the original entries from the 0.11 state.

$ tf state show module.sunset_environment.aws_lb_target_group.arthr | grep arn
id                                 = arn:aws:elasticloadbalancing:eu-west-1:000000000000:targetgroup/dev-dev2-arthr/55ccc73b8681221d
arn                                = arn:aws:elasticloadbalancing:eu-west-1:000000000000:targetgroup/dev-dev2-arthr/55ccc73b8681221d
arn_suffix                         = targetgroup/dev-dev2-arthr/55ccc73b8681221d

References

None

apparentlymart commented 5 years ago

Hi @npc-adrian! Sorry for this strange behavior and thanks for reporting it.

That error message does indeed seem to come from the AWS provider -- or rather, from the underlying AWS API itself -- so I think we'll need some folks with AWS provider expertise to look at this to understand what's going on here. Would you mind opening an issue for this in the AWS provider repository to start, and then we can reopen this here if the AWS provider team thinks there's an upstream Terraform Core problem to be fixed? Thanks!

npc-adrian commented 5 years ago

Hi @apparentlymart. Yeah, that's fine. I'll put it on their issues list

ghost commented 5 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform