search

aws / containers-roadmap

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).
https://aws.amazon.com/about-aws/whats-new/containers/
Other
5.21k stars 319 forks source link

[ECS]: Add ecs:ListTagsForResource permission to ECS managed instance role: AmazonEC2ContainerServiceforEC2Role #1979

Open colinbjohnson opened 1 year ago

colinbjohnson commented 1 year ago

Summary

The AWS ECS Agent makes a ecs:ListTagsForResource call that is not allowed by the arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role role. It is expected that the arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role and AWS ECS Agent be "in sync" with each other in regards to provided rights and used rights.

Description

When running an ECS Cluster where the EC2 Instances utilize the arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role IAM Role errors are frequently produced both within CloudTrail and within the AWS ECS Agent logs as a result of the missing permission.

Expected Behavior

It is expected that the AWS ECS Agent and arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role role remain in sync in terms of provided rights and used rights and that the error shown below does not occur:

Task Metadata error: unable to get 'TaskTags' for 'arn:aws:ecs:us-west-2:123456789012:task/dev-cluster/7f922e40856147e0b7a7b07cdf90ba92': AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/ECSClusterNode-dev/i-0123456789abcdef0 is not authorized to perform: ecs:ListTagsForResource on resource: arn:aws:ecs:us-west-2:123456789012:task/dev-cluster/7f922e40856147e0b7a7b07cdf90ba92 because no identity-based policy allows the ecs:ListTagsForResource action

Observed Behavior

Both CloudTrail and Amazon ECS Logs fill with the following error:

Task Metadata error: unable to get 'TaskTags' for 'arn:aws:ecs:us-west-2:123456789012:task/dev-cluster/7f922e40856147e0b7a7b07cdf90ba92': AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/ECSClusterNode-dev/i-0123456789abcdef0 is not authorized to perform: ecs:ListTagsForResource on resource: arn:aws:ecs:us-west-2:123456789012:task/dev-cluster/7f922e40856147e0b7a7b07cdf90ba92 because no identity-based policy allows the ecs:ListTagsForResource action

Environment Details

We are running Amazon ECS AMI ami-02c6f7952af6bd632 running Amazon ECS Agent - v1.61.3 (63f97f40) (linux).

Supporting Log Snippets

Realmonia commented 1 year ago

Hi @colinbjohnson , AmazonEC2ContainerServiceforEC2Role role does not contain a policy that allows ecs:ListTagsForResource. Can you add this policy to your ECSInstanceRole so the problem is mitigated?

Besides, can you please comment on why

It is expected that the arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role and AWS ECS Agent be "in sync" with each other in regards to provided rights and used rights.

I am asking because I don't think we have documentation saying that having AmazonEC2ContainerServiceforEC2Role is sufficient to use all ECS agent features. Your input will help us improving our documentation.

A little bit more context: only taskWithTags endpoint in task metadata endpoint v3/v4 and taskWithTags(metadataWithTags in code) endpoint endpoint in task metadata endpoint v2 are the only endpoint/version combinations for all task metadata endpoints that actually interact with ECS control plane, and their current implementation is using the ECS client created with credential provider (default to ecsInstanceRole).

I will ask our doc team to update the required access in task metadata endpoint documentation.

kevioke commented 1 year ago

I'm running into a similar problem. I'm not sure what changed recently, but some of our clusters are encountering this ecs:ListTagsForResource access error as well as throttling on the same API call, most likely because the agent is retrying. @Realmonia the documentation does state that the AmazonEC2ContainerServiceforEC2Role supports the full Amazon ECS feature set.

Amazon ECS provides the AmazonEC2ContainerServiceforEC2Role managed IAM policy which contains the permissions needed to use the full Amazon ECS feature set. This managed policy can be attached to an IAM role and associated with your container instances. Alternatively, you can use the managed policy as a guide when creating a custom policy to use. The container instance role provides permissions needed for the Amazon ECS container agent and Docker daemon to call AWS APIs on your behalf. For more information on the managed policy, see AmazonEC2ContainerServiceforEC2Role.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html

sparrc commented 1 year ago

I have submitted a request to our docs team to update this doc with the information that ecs.ListTagsForResource is required to use the taskWithTags endpoint for now: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4.html

Will keep this issue open as a feature request in the containers-roadmap repo to add ListTagsForResource to the default IAM role.