search

hashicorp / terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
https://registry.terraform.io/providers/hashicorp/aws
Mozilla Public License 2.0
9.83k stars 9.18k forks source link

[Bug]: Release of 5.73.0 leading to change in required IAM permission #40104

Open kentedstrom opened 1 day ago

kentedstrom commented 1 day ago

Terraform Core Version

1.9.8

AWS Provider Version

5.73.0

Affected Resource(s)

Affected Resource(s): RDS Database: The issue occurred during the attempt to provision an RDS instance, although it affected all infrastructure provisioning tasks. EC2 (DescribeTags): The error specifically referenced the ec2:DescribeTags action, which was blocked due to insufficient IAM permissions. This action is part of the EC2 API and is required during infrastructure provisioning, especially when describing resources and tags. IAM Role/Policy: The IAM role we utilize was impacted by changes in required permissions following the new release of the Terraform AWS provider. This role and its perms have not been changed and were working fine before 5.73.0

Expected Behavior

Successful Infrastructure Provisioning: Terraform should have successfully provisioned all infrastructure, including the RDS database, and any other resources defined in the configuration, without encountering authorization errors.

Appropriate IAM Permissions: The IAM role used by Terraform should have had the necessary permissions to perform all required actions, including ec2:DescribeTags and any other related permissions needed for the provisioning of resources across environments. This role has not been changed and we tested reverting back to 5.72.1 without the EC2(DescribeTags) permission and it worked as intended.

Backward Compatibility: The Terraform AWS provider should have been backward compatible, meaning that the new version (5.73.0) should not have introduced any breaking changes or new IAM permission requirements that would block deployments using previously valid configurations.

Actual Behavior

Provisioning Failed Due to Permission Errors: The infrastructure provisioning failed across all environments when trying to create resources, including the RDS database. The error message specifically indicated that the IAM role assumed by Terraform which was not authorized to perform the ec2:DescribeTags action due to missing identity-based policy permissions. This permission was not required before this version.

We verified the issue was with 5.73.0 by reverting back to 5.72.1 without the ec2:DescribeTags perm and it worked as intended.

Relevant Error/Panic Output Snippet

#1:
Error: listing tags for EC2 (Elastic Compute Cloud) <thing> (subnet-foobar): operation error EC2: DescribeTags, https response error StatusCode: 403, RequestID: 813ab14b-56d7-4b4b-958c-47a9dce4f1ef, api error UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::foobar:assumed-role/foobar/foobar is not authorized to perform: ec2:DescribeTags because no identity-based policy allows the ec2:DescribeTags action
with module.subnets.data.aws_subnet.all["subnet-foobar],
on .terraform/modules//main.tf line 32, in data "aws_subnet" "all":
32: data "aws_subnet" "all" {
Error: listing tags for EC2 (Elastic Compute Cloud) <thing> (subnet-foobar): operation error EC2: DescribeTags, https response error StatusCode: 403, RequestID: 69607a85-5728-459a-9a64-c6cc920aee85, api error UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::foobar:assumed-role/foobar/aws-go-sdk-foobar is not authorized to perform: ec2:DescribeTags because no identity-based policy allows the ec2:DescribeTags action
with module.subnets.data.aws_subnet.all["subnet-foobar"],
on .terraform/modules/subnets/main.tf line 32, in data "aws_subnet" "all":
32: data "aws_subnet" "all" {
Error: listing tags for EC2 (Elastic Compute Cloud) <thing> (subnet-foobar): operation error EC2: DescribeTags, https response error StatusCode: 403, RequestID: b4847c91-999d-4f4f-ac71-9e015da9f602, api error UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::foobar:assumed-role/foobar/aws-go-sdk-foobar is not authorized to perform: ec2:DescribeTags because no identity-based policy allows the ec2:DescribeTags action
with module.subnets.data.aws_subnet.all["subnet-foobar"],
on .terraform/modules/subnets/main.tf line 32, in data "aws_subnet" "all":
32: data "aws_subnet" "all" {

Terraform Configuration Files

data "aws_availability_zones" "available" { state = "available" }

All data subnets for this VPC and Cluster

data "aws_subnets" "subnets" { filter { name = "vpc-id" values = [var.vpc_id] }

tags = { "riot.io/resource-type/data" = true "riot.io/cluster-name" = local.cluster_name } }

All data subnets for this VPC and Cluster with detailed information such as available IPs

data "aws_subnet" "all" { for_each = toset(data.aws_subnets.subnets.ids) id = each.key }

Steps to Reproduce

  1. Ensure Terraform is Installed

  2. Use an IAM Role Without Necessary Permissions: Set up an IAM role with the following permissions (or lack thereof):

    The role should have limited permissions (specifically missing ec2:DescribeTags and other necessary permissions for resource provisioning). This IAM role should be assumed by Terraform during infrastructure provisioning.

  3. Create or Use Existing Terraform Configuration: Create a Terraform configuration that provisions resources across different AWS services (e.g., EC2, RDS, etc.). Include resources that rely on ec2:DescribeTags or similar EC2 API actions.

  4. Configure Terraform to Use the AWS Provider: Set up the AWS provider configuration in Terraform, ensuring that you’re using a version of the AWS provider prior to the update (or ensure Terraform automatically pulls the latest version).

  5. Run Terraform Plan/Apply: Execute the terraform apply or terraform plan command, allowing Terraform to provision resources using the IAM role from step 2.

  6. Observe the Error: Upon running the terraform apply or terraform plan command, observe the error message similar to:

UnauthorizedOperation: You are not authorized to perform this operation. 
User: arn:aws:sts::777474137401:assumed-role/blah is not authorized to perform: 
ec2:DescribeTags because no identity-based policy allows the ec2:DescribeTags action.
Confirm Permission Issue: Verify that the IAM role being used does not have the ec2:DescribeTags permission or other permissions introduced by the new AWS provider version (5.73.0 or higher), which prevents successful resource provisioning.

Debug Output

No response

Panic Output

No response

Important Factoids

  1. We are utilizing an in-house IAC application but are confident that it's not a contributing factor
  2. We tested reverting to version 5.72.1 and that fixed the issue even WITHOUT the additional IAM permission of ec2:describetags

References

We suspect this commit may be a cause: https://github.com/hashicorp/terraform-provider-aws/commit/802b6329bf0e12f2d9bfd4180141c2598a09c84e

Would you like to implement a fix?

No

github-actions[bot] commented 1 day ago

Community Note

Voting for Prioritization

Volunteering to Work on This Issue

novekm commented 5 hours ago

@justinretzolk can you take a look into this when you have some time?