search

grailbio / reflow

A language and runtime for distributed, incremental data processing in the cloud
Apache License 2.0
965 stars 52 forks source link

Example AWS IAM role policy for Reflow #99

Closed olgabot closed 5 years ago

olgabot commented 5 years ago

Hello, I'm struggling to get a fresh instance up and running with reflow. There's some issue with my IAM permissions and I'm struggling to figure out how to fix them.

I currently have these policies:

#### AmazonS3FullAccess ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:*", "Resource": "*" } ] } ``` #### AmazonAPIGatewayPushToCloudWatchLogs ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:PutLogEvents", "logs:GetLogEvents", "logs:FilterLogEvents" ], "Resource": "*" } ] } ``` #### AmazonEC2ContainerRegistryPowerUser ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecr:GetAuthorizationToken", "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer", "ecr:GetRepositoryPolicy", "ecr:DescribeRepositories", "ecr:ListImages", "ecr:DescribeImages", "ecr:BatchGetImage", "ecr:InitiateLayerUpload", "ecr:UploadLayerPart", "ecr:CompleteLayerUpload", "ecr:PutImage" ], "Resource": "*" } ] } ``` #### IAMReadOnlyAccess ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iam:GenerateCredentialReport", "iam:GenerateServiceLastAccessedDetails", "iam:Get*", "iam:List*", "iam:SimulateCustomPolicy", "iam:SimulatePrincipalPolicy" ], "Resource": "*" } ] } ``` #### AmazonElasticFileSystemFullAccess ``` { "Version": "2012-10-17", "Statement": [ { "Action": [ "ec2:CreateNetworkInterface", "ec2:DeleteNetworkInterface", "ec2:DescribeAvailabilityZones", "ec2:DescribeNetworkInterfaceAttribute", "ec2:DescribeNetworkInterfaces", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVpcAttribute", "ec2:DescribeVpcs", "ec2:ModifyNetworkInterfaceAttribute", "elasticfilesystem:*", "kms:DescribeKey", "kms:ListAliases" ], "Effect": "Allow", "Resource": "*" } ] } ``` #### AmazonEC2ContainerServiceforEC2Role ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecs:CreateCluster", "ecs:DeregisterContainerInstance", "ecs:DiscoverPollEndpoint", "ecs:Poll", "ecs:RegisterContainerInstance", "ecs:StartTelemetrySession", "ecs:UpdateContainerInstancesState", "ecs:Submit*", "ecr:GetAuthorizationToken", "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "*" } ] } ```

Before I added the AmazonElasticFileSystemFullAccess and AmazonEC2ContainerServiceforEC2Role policies, I got the error below.

``` ➜ minirun git:(master) ✗ reflow runbatch reflow: batch program /home/ubuntu/reflow-workflows/workflows/rnaseq.rf runsfile samples.csv reflow: ec2cluster: instance launch error: AccessDeniedException: User: arn:aws:sts::423543210473:assumed-role/S3fromEC2/i-0fc7f464fcbc169df is not authorized to perform: ecr:GetAuthorizationToken on resource: * status code: 400, request id: 742809e4-02e8-11e9-9119-0dd157128f95 reflow: ec2cluster: error while launching instance: AccessDeniedException: User: arn:aws:sts::423543210473:assumed-role/S3fromEC2/i-0fc7f464fcbc169df is not authorized to perform: ecr:GetAuthorizationToken on resource: * status code: 400, request id: 742809e4-02e8-11e9-9119-0dd157128f95 reflow: ec2cluster: instance launch error: AccessDeniedException: User: arn:aws:sts::423543210473:assumed-role/S3fromEC2/i-0fc7f464fcbc169df is not authorized to perform: ecr:GetAuthorizationToken on resource: * status code: 400, request id: 74280a23-02e8-11e9-8773-0d770e495077 reflow: ec2cluster: error while launching instance: AccessDeniedException: User: arn:aws:sts::423543210473:assumed-role/S3fromEC2/i-0fc7f464fcbc169df is not authorized to perform: ecr:GetAuthorizationToken on resource: * ... more of the same ... ```

Now, I added the EC2 policies and I still get an UnauthorizedOperation error. Do you know which policies should be active for Reflow to work?

➜  minirun git:(master) ✗ reflow runbatch -retry
reflow: batch program /home/ubuntu/reflow-workflows/workflows/rnaseq.rf runsfile samples.csv
retrying run OPS016_mBAL_RNA_246_L11_S133
retrying run OPS016_mBAL_RNA_229_L9_S132
retrying run OPS016_mBAL_RNA_234_P9_S148
retrying run OPS016_mBAL_RNA_240_H11_S118
retrying run OPS016_mBAL_RNA_241_J11_S125
retrying run OPS016_mBAL_RNA_232_N9_S140
retrying run OPS016_mBAL_RNA_235_B11_S97
retrying run OPS016_mBAL_RNA_237_D11_S104
retrying run OPS016_mBAL_RNA_239_F11_S111
reflow: ec2cluster: instance launch error: UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: -6lBVppOemjcS2H-0WZaMH311uKoNhKQ1Hb05ftMCZ3efU1cISuwPaUygbvFrYo4wFnsG1h7SFkL_6cr4s6BUnDbqfY4r52kcxUCKSoUKRbxeKP4-h70b_GbPAq7-GDjOjBc5QJqoUs1LmyxuUMTMkabR1ZFI-lvZ6FiXAYptixagnHy2it74r84qIIRKFO1Zak-NX_LoqG3x1UuxoS1TY25BLSsMXXJMWMEEM484NyQs_m7Aeha9wvdEgEiVWgP5LygB8xtopZvRnKt3C7CqAzh_Vw0Y_eDPIXBo3iWz4z_kdKheKVQGviR9bINwFmPh1vv9oo7mIp2Mi8jsgWnNQHljAXEFuhycAvMpMhCihjxR9O5gL_0weRDzZ0KtyPj6_Bl-nsymbWEIGo6g5mPjIMsNDF88XHc7NJjNJ1DUQc7lQp6CRp2LrjwMVjuELEqZ6mnW_qQWmukgSNgGMAbwDWIv_W5SRp0M9ABfK623Vi6BU-E1I5rA-ZLt9spzTkHd5efdrfXvBq7YYoex8NcjvVHnj8I2eIBWF53mK74M2Yig02woxcbNOhUE6v28yz5xBzyxBPsvGxnXzUANJjK54uqd_OwB9lvzj27lTmorVx6FBVDxSq2QXS52MKyw5DvGFSRmB8pXT2UG_utvA9bc_SVTeCflxX3wAuZXUlly8lLvGm-4z2MrUHM5nD-Pu3VeHS-dY1AcCo
        status code: 403, request id: d4ee077c-2b12-4c2b-a62b-060b0f04879a
reflow: ec2cluster: error while launching instance: UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: -6lBVppOemjcS2H-0WZaMH311uKoNhKQ1Hb05ftMCZ3efU1cISuwPaUygbvFrYo4wFnsG1h7SFkL_6cr4s6BUnDbqfY4r52kcxUCKSoUKRbxeKP4-h70b_GbPAq7-GDjOjBc5QJqoUs1LmyxuUMTMkabR1ZFI-lvZ6FiXAYptixagnHy2it74r84qIIRKFO1Zak-NX_LoqG3x1UuxoS1TY25BLSsMXXJMWMEEM484NyQs_m7Aeha9wvdEgEiVWgP5LygB8xtopZvRnKt3C7CqAzh_Vw0Y_eDPIXBo3iWz4z_kdKheKVQGviR9bINwFmPh1vv9oo7mIp2Mi8jsgWnNQHljAXEFuhycAvMpMhCihjxR9O5gL_0weRDzZ0KtyPj6_Bl-nsymbWEIGo6g5mPjIMsNDF88XHc7NJjNJ1DUQc7lQp6CRp2LrjwMVjuELEqZ6mnW_qQWmukgSNgGMAbwDWIv_W5SRp0M9ABfK623Vi6BU-E1I5rA-ZLt9spzTkHd5efdrfXvBq7YYoex8NcjvVHnj8I2eIBWF53mK74M2Yig02woxcbNOhUE6v28yz5xBzyxBPsvGxnXzUANJjK54uqd_OwB9lvzj27lTmorVx6FBVDxSq2QXS52MKyw5DvGFSRmB8pXT2UG_utvA9bc_SVTeCflxX3wAuZXUlly8lLvGm-4z2MrUHM5nD-Pu3VeHS-dY1AcCo
        status code: 403, request id: d4ee077c-2b12-4c2b-a62b-060b0f04879a
reflow: ec2cluster: instance launch error: UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: TLCssOYNQG-lZnMKYYXuabYdsNPm6eftFeP1knaZKPPkF2HdgHXzbHzL216Cqy9APbPgasw7dQZvAFf1zgipdcwytKHOcWRUnlo-f_fT0jXZCvZHnVG4Z6qIj4y9KOvwBP2DAPRR-60EBFhozmIC1oieePIAjmP6F5bLpVbnfQMzkVCjKh1nzskmAkNMUEsg7C4-l01Smxd-5q49WjbQm8fckBgDjwlomXs8FwrBAq03hEYYritRtF2raq97ktZ65GOmsTlGP0N0XkWdnf2GFQkPt4VTct-UkfuT707VV1b7I8Pbrl2qiYwxbn7x5oJrdhX1K0W_iRytFwQvMnT9z1_RJO2oA5AD311DwisqXQE3jOXT1xRSGBVcb-Cs874tw7PD0RxY2FXs4B_y71Uak3EOfFd3TjTn49c0tlEYQtaSp9A3EtIQXPS_7IOoOsG1t5R0Ng_nwFm21xlk95O7F9SvT38Be3OIMLaZnKpA0U3Ujffx7QPg_WRJR6rSOjCSYlGcw2e69JFXEkCIj5BBgJ7LxZvFDKsFxRmgUFPJn0gdvkmQU3Jlg_rEud88F2Icl8498G38rP8i63X1PuKfeN-NpobfMPXdwoAYom-rsJ3MEM455DMDGrJlgLqmLzjC2HwZth_GNHP8AOShFmHp9UhpKCdwFEaLOpFrgm2NHvUamyzY74Dsz0rGPOT1XZ0X2y7SnEBStgA
... more of the same ...
olgabot commented 5 years ago

SOLVED: I also needed AmazonEC2FullAccess

AmazonEC2FullAccess

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "ec2:*",
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "elasticloadbalancing:*",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "cloudwatch:*",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "autoscaling:*",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:AWSServiceName": [
                        "autoscaling.amazonaws.com",
                        "ec2scheduled.amazonaws.com",
                        "elasticloadbalancing.amazonaws.com",
                        "spot.amazonaws.com",
                        "spotfleet.amazonaws.com",
                        "transitgateway.amazonaws.com"
                    ]
                }
            }
        }
    ]
}