search

awslabs / service-workbench-on-aws

A platform that provides researchers with one-click access to collaborative workspace environments operating across teams, universities, and datasets while enabling university IT stakeholders to manage, monitor, and control spending, apply security best practices, and comply with corporate governance.
Apache License 2.0
177 stars 119 forks source link

cfn template "emr-cluster.cfn.yml" and related resource should be modified. #123

Closed ghost closed 3 years ago

ghost commented 4 years ago

deployment region - ap-northeast-2

1. In the template, Service Role for EMR does not have KMS policy. Thus SecurityConfiguration makes below problem.

The EMR Service Role must have the kms:GenerateDataKey and kms:ReEncrypt permission for the KMS key configuration when you enabled EBS encryption by default. You can retrieve that KMS key's ID by using the ec2:GetEbsDefaultKmsKeyId API.

KMS policy should be added to the Service Role. Identified as my member account setting problem

  1. Fail to bootstrap "us-east-1.elasticmapreduce/bootstrap-actions/run-if" on non us-east-1 region. EMRFS could not download bootstrap file on other region. Thus it makes below problem.

Master instance failed attempting to download bootstrap action 1 file from S3

run-if script should be copied to the same region bucket and cfn template should be modified.

3. Additionally, cross account role also should have KMS policy. Identified as my member account setting problem

rsmayda commented 4 years ago

Hey are you still facing any issues? I see that 1 & 3 seem to be account setting issues. Is 2 still applicable?

ghost commented 4 years ago

Could you please make EMR cluster on the other region except us-east-1?

rsmayda commented 4 years ago

Thanks for confirming issue 2 is still present -- we will keep this issue up to date with our progress

SanketD92 commented 4 years ago

@inhwan-aws Thanks for bringing this up. We have identified the potential root cause for this bug. The bootstrap action script path and IAM policy resource reference the us-east-1 region in the emr templates (built-in, service-catalog and external), which would need the following changes:

From
s3://us-east-1.elasticmapreduce/bootstrap-actions/run-if
to
!Sub s3://${AWS::Region}.elasticmapreduce/bootstrap-actions/run-if
From
'arn:aws:s3:::us-east-1.elasticmapreduce/bootstrap-actions/run-if'
to
!Sub 'arn:aws:s3:::${AWS::Region}.elasticmapreduce/bootstrap-actions/run-if'

We will be making and testing these changes soon. Stay tuned.

ghost commented 4 years ago

Thank you, Sanket!

virgo81net commented 4 years ago

@SanketD92

In the process of combining the two accounts, the existing account (@inhwan-aws) was deleted, so please understand that previous posts and comments appear as ghost :)

I tried what you suggest but a bucket "elasticmapreduce" must be unique globally, so it doesn't exist in other regions. Thus, this solution is not work for me. (my region is ap-northeast-2)

From
s3://us-east-1.elasticmapreduce/bootstrap-actions/run-if
to
!Sub s3://${AWS::Region}.elasticmapreduce/bootstrap-actions/run-if
From
'arn:aws:s3:::us-east-1.elasticmapreduce/bootstrap-actions/run-if'
to
!Sub 'arn:aws:s3:::${AWS::Region}.elasticmapreduce/bootstrap-actions/run-if'

Thanks and regards, Inhwan

jn1119 commented 3 years ago

Hi @virgo81net

The patch that was recommended here worked for me and has been merged now. Did you import the latest updated Service Catalog product for EMR that would have reflected the latest template?

Thanks

jn1119 commented 3 years ago

Fixed in PR https://github.com/awslabs/service-workbench-on-aws/pull/394