AccessDenied: User: arn:aws:iam::xxx:user/xxx is not authorized to perform: sts:AssumeRole on resource

jgorgel commented 10 months ago

$ copilot env deploy --name prod ✘ get template version of environment prod: get metadata for stack chatbot-prod: get template summary: AccessDenied: User: arn:aws:iam::xxx:user/xxx is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::123123:role/chatbot-prod-EnvManagerRole status code: 403, request id: 5c5abff6-30d7-4cd3-adf7-b16c474a3982

AWS CLI/SDK Version(s): copilot-cli version: v1.32.0, built for linux

Lou1415926 commented 10 months ago

Can you try adding the permission to assume the EnvManagerRole for the user arn:aws:iam::xxx:user/xxx? The statement should look something like:

    - Effect: Allow
      Action:
        - "sts:AssumeRole"
      Resource:
        - "arn:aws:iam::123123:role/chatbot-prod-EnvManagerRole"

Copilot doesn't automatically add this permission to the user, but it does automatically allow the root user (i.e. arn:aws:iam::123123:root) to assume the env manager role.

jgorgel commented 10 months ago

Thanks @Lou1415926. There is no chatbot-prod-EnvManagerRole, I guess it has been deleted.

Can i recreate my prod env totally?

Lou1415926 commented 10 months ago

Yeah that's totally possible! Can you go to the CloudFormation stack console (of the correct region) and see if the chatbot-prod stack is there? I can suggest a way to delete & recreate the environment from this point.

jgorgel commented 10 months ago

yes, there is a chatbot-prod stack. How can i recreate my prod env?

Lou1415926 commented 10 months ago

In this case let's try creating a "chatbot-prod-EnvManagerRole", and then run copilot env delete to let Copilot handle all the fuss.

Go to the IAM console, create a new role that's called exactly "chatbot-prod-EnvManagerRole".

Use ⬇️ as the "AssumeRolePolicy" (or select "AWS account" for "Trusted entity" and then enter your account)

{
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::<your account number>:root"
        },
        "Action": "sts:AssumeRole"
    }
]
}

Give the temporary prod role an admin permission to make it easier. Alternatively, try ⬇️ permissions. These are roughly the typical permissions an environment manager role have -

{
"Version": "2012-10-17",
"Statement": [
    {
        "Action": [
            "logs:GetLogRecord",
            "logs:GetQueryResults",
            "logs:StartQuery",
            "logs:GetLogEvents",
            "logs:DescribeLogStreams",
            "logs:StopQuery",
            "logs:TestMetricFilter",
            "logs:FilterLogEvents",
            "logs:GetLogGroupFields",
            "logs:GetLogDelivery"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "CloudwatchLogs"
    },
    {
        "Action": [
            "cloudwatch:DescribeAlarms"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "Cloudwatch"
    },
    {
        "Action": [
            "ecs:ListAttributes",
            "ecs:ListTasks",
            "ecs:DescribeServices",
            "ecs:DescribeTaskSets",
            "ecs:ListContainerInstances",
            "ecs:DescribeContainerInstances",
            "ecs:DescribeTasks",
            "ecs:DescribeClusters",
            "ecs:UpdateService",
            "ecs:PutAttributes",
            "ecs:StartTelemetrySession",
            "ecs:StartTask",
            "ecs:StopTask",
            "ecs:ListServices",
            "ecs:ListTaskDefinitionFamilies",
            "ecs:DescribeTaskDefinition",
            "ecs:ListTaskDefinitions",
            "ecs:ListClusters",
            "ecs:RunTask",
            "ecs:ListServicesByNamespace"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "ECS"
    },
    {
        "Condition": {
            "StringEquals": {
                "aws:ResourceTag/copilot-environment": "prod",
                "aws:ResourceTag/copilot-application": "chatbot"
            }
        },
        "Action": [
            "ecs:ExecuteCommand",
            "ssm:StartSession"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "ExecuteCommand"
    },
    {
        "Action": [
            "states:StartExecution",
            "states:DescribeStateMachine"
        ],
        "Resource": [
            "arn:aws:states:us-west-2:568623488001:stateMachine:chatbot-prod-*"
        ],
        "Effect": "Allow",
        "Sid": "StartStateMachine"
    },
    {
        "Action": [
            "cloudformation:CancelUpdateStack",
            "cloudformation:CreateChangeSet",
            "cloudformation:CreateStack",
            "cloudformation:DeleteChangeSet",
            "cloudformation:DeleteStack",
            "cloudformation:Describe*",
            "cloudformation:DetectStackDrift",
            "cloudformation:DetectStackResourceDrift",
            "cloudformation:ExecuteChangeSet",
            "cloudformation:GetTemplate",
            "cloudformation:GetTemplateSummary",
            "cloudformation:UpdateStack",
            "cloudformation:UpdateTerminationProtection"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "CloudFormation"
    },
    {
        "Condition": {
            "StringEquals": {
                "iam:ResourceTag/copilot-application": "chatbot",
                "iam:ResourceTag/copilot-environment": "prod"
            }
        },
        "Action": [
            "iam:GetRole",
            "iam:PassRole"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "GetAndPassCopilotRoles"
    },
    {
        "Action": [
            "ecr:BatchGetImage",
            "ecr:BatchCheckLayerAvailability",
            "ecr:CompleteLayerUpload",
            "ecr:DescribeImages",
            "ecr:DescribeRepositories",
            "ecr:GetDownloadUrlForLayer",
            "ecr:InitiateLayerUpload",
            "ecr:ListImages",
            "ecr:ListTagsForResource",
            "ecr:PutImage",
            "ecr:UploadLayerPart",
            "ecr:GetAuthorizationToken"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "ECR"
    },
    {
        "Action": [
            "resource-groups:GetGroup",
            "resource-groups:GetGroupQuery",
            "resource-groups:GetTags",
            "resource-groups:ListGroupResources",
            "resource-groups:ListGroups",
            "resource-groups:SearchResources"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "ResourceGroups"
    },
    {
        "Action": [
            "ssm:DeleteParameter",
            "ssm:DeleteParameters",
            "ssm:GetParameter",
            "ssm:GetParameters",
            "ssm:GetParametersByPath"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "SSM"
    },
    {
        "Action": [
            "ssm:PutParameter",
            "ssm:AddTagsToResource"
        ],
        "Resource": [
            "arn:aws:ssm:us-west-2:568623488001:parameter/copilot/chatbot/prod/secrets/*"
        ],
        "Effect": "Allow",
        "Sid": "SSMSecret"
    },
    {
        "Action": [
            "ssm:StartSession"
        ],
        "Resource": [
            "arn:aws:ssm:us-west-2::document/AWS-StartPortForwardingSessionToRemoteHost"
        ],
        "Effect": "Allow",
        "Sid": "SSMSession"
    },
    {
        "Action": [
            "elasticloadbalancing:DescribeLoadBalancerAttributes",
            "elasticloadbalancing:DescribeSSLPolicies",
            "elasticloadbalancing:DescribeLoadBalancers",
            "elasticloadbalancing:DescribeTargetGroupAttributes",
            "elasticloadbalancing:DescribeListeners",
            "elasticloadbalancing:DescribeTags",
            "elasticloadbalancing:DescribeTargetHealth",
            "elasticloadbalancing:DescribeTargetGroups",
            "elasticloadbalancing:DescribeRules"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "ELBv2"
    },
    {
        "Action": [
            "s3:ListBucketByTags",
            "s3:GetLifecycleConfiguration",
            "s3:GetBucketTagging",
            "s3:GetInventoryConfiguration",
            "s3:GetObjectVersionTagging",
            "s3:ListBucketVersions",
            "s3:GetBucketLogging",
            "s3:ListBucket",
            "s3:GetAccelerateConfiguration",
            "s3:GetBucketPolicy",
            "s3:GetObjectVersionTorrent",
            "s3:GetObjectAcl",
            "s3:GetEncryptionConfiguration",
            "s3:GetBucketRequestPayment",
            "s3:GetObjectVersionAcl",
            "s3:GetObjectTagging",
            "s3:GetMetricsConfiguration",
            "s3:HeadBucket",
            "s3:GetBucketPublicAccessBlock",
            "s3:GetBucketPolicyStatus",
            "s3:ListBucketMultipartUploads",
            "s3:GetBucketWebsite",
            "s3:ListJobs",
            "s3:GetBucketVersioning",
            "s3:GetBucketAcl",
            "s3:GetBucketNotification",
            "s3:GetReplicationConfiguration",
            "s3:ListMultipartUploadParts",
            "s3:GetObject",
            "s3:GetObjectTorrent",
            "s3:GetAccountPublicAccessBlock",
            "s3:ListAllMyBuckets",
            "s3:DescribeJob",
            "s3:GetBucketCORS",
            "s3:GetAnalyticsConfiguration",
            "s3:GetObjectVersionForReplication",
            "s3:GetBucketLocation",
            "s3:GetObjectVersion",
            "kms:Decrypt"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "BuiltArtifactAccess"
    },
    {
        "Action": [
            "ec2:DescribeSubnets",
            "ec2:DescribeSecurityGroups",
            "ec2:DescribeNetworkInterfaces",
            "ec2:DescribeRouteTables"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "EC2"
    },
    {
        "Action": [
            "apprunner:DescribeService",
            "apprunner:ListOperations",
            "apprunner:ListServices",
            "apprunner:PauseService",
            "apprunner:ResumeService",
            "apprunner:StartDeployment",
            "apprunner:DescribeObservabilityConfiguration",
            "apprunner:DescribeVpcIngressConnection"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "AppRunner"
    },
    {
        "Action": [
            "tag:GetResources"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "Tags"
    },
    {
        "Action": [
            "application-autoscaling:DescribeScalingPolicies"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "ApplicationAutoscaling"
    },
    {
        "Action": [
            "iam:DeleteRole",
            "iam:ListRolePolicies",
            "iam:DeleteRolePolicy"
        ],
        "Resource": [
            "arn:aws:iam::568623488001:role/chatbot-prod-CFNExecutionRole",
            "arn:aws:iam::568623488001:role/chatbot-prod-EnvManagerRole"
        ],
        "Effect": "Allow",
        "Sid": "DeleteRoles"
    },
    {
        "Action": [
            "cloudformation:DescribeStacks",
            "cloudformation:DeleteStack"
        ],
        "Resource": [
            "arn:aws:cloudformation:us-west-2:568623488001:stack/chatbot-prod/*"
        ],
        "Effect": "Allow",
        "Sid": "DeleteEnvStack"
    },
    {
        "Action": [
            "rds:DescribeDBInstances",
            "rds:DescribeDBClusters"
        ],
        "Resource": "*",
        "Effect": "Allow",
        "Sid": "RDS"
    }
]
}

Run copilot svc delete --env prod for api-backend, celery-worker, and any other services that you've deployed to "prod". For "celery-worker", you might need to manually delete it from the console since it seems to be stuck in the "REVIEW_IN_PROGRESS" state.
Run copilot env delete --name prod to delete "prod".

jgorgel commented 10 months ago

Thanks @Lou1415926. After manually created chatbot-prod-EnvManagerRole and chatbot-prod-CFNExecutionRole, it fixed following the steps you gave.

Lou1415926 commented 10 months ago

I will close the issue for now. Feel free to reopen it if you have further questions!

aws / copilot-cli

AccessDenied: User: arn:aws:iam::xxx:user/xxx is not authorized to perform: sts:AssumeRole on resource #5615