awslabs / service-workbench-on-aws

A platform that provides researchers with one-click access to collaborative workspace environments operating across teams, universities, and datasets while enabling university IT stakeholders to manage, monitor, and control spending, apply security best practices, and comply with corporate governance.
Apache License 2.0
177 stars 119 forks source link

[Bug] EMR creation error in some regions #1084

Open sungeuns opened 1 year ago

sungeuns commented 1 year ago

Describe the bug

When user using SWB in some region (for example ap-northeast-2) EMR creation is failed. Basically EMR use 3 bootstrap script in CloudFormation template. Third one is Log4j path script. It is located us-east-1 in some region like us-west-2 it works but some region like ap-northeast-2 cannot download this script.

But log4j script download path is fixed, user should upload log4j patch script their own bucket and change it to successfully create EMR.

I think two part should be changed. EMR CF template use fixed log4j script path here and use role here

To fix this issue, need to add log4j script to user's bucket or need to upload script in common bucket for each region and specify this bucket.

To Reproduce Steps to reproduce the behavior:

  1. Deploy SWB in some region like ap-northeast-2
  2. Post deployment setup (Make AMI using SDC)
  3. Create EMR workspace

Expected behavior No errors on creating EMR, but there's bootstrap action failed error like below:

emr-error-log

Error provisioning environment emr8. Reason: Errors from CloudFormation: [{LogicalResourceId : SC-117579184231-pp-dcyb4nvdu4pgq, ResourceType : AWS::CloudFormation::Stack, StatusReason : The following resource(s) failed to create: [EmrCluster]. Rollback requested by user.}, {LogicalResourceId : EmrCluster, ResourceType : AWS::EMR::Cluster, StatusReason : ElasticMapReduce Cluster with Id j-3PAV66YLYFK3V, is in state TERMINATING and failed to stabilize due to the following reason: {Code: BOOTSTRAP_FAILURE,Message: Master instance (i-09e9438319a7d6db9) failed attempting to download bootstrap action 3 file from S3}}, {LogicalResourceId : EmrCluster, ResourceType : AWS::EMR::Cluster, StatusReason : Resource creation Initiated}, {LogicalResourceId : InstanceProfile, ResourceType : AWS::IAM::InstanceProfile, StatusReason : Resource creation Initiated}, {LogicalResourceId : Ec2Role, ResourceType : AWS::IAM::Role, StatusReason : Resource creation Initiated}, {LogicalResourceId : LogBucketPolicy, ResourceType : AWS::S3::BucketPolicy, StatusReason : Resource creation Initiated}, {LogicalResourceId : MasterSecurityGroup, ResourceType : AWS::EC2::SecurityGroup, StatusReason : Resource creation Initiated}, {LogicalResourceId : InstanceRolePermissionBoundary, ResourceType : AWS::IAM::ManagedPolicy, StatusReason : Resource creation Initiated}, {LogicalResourceId : EmrSecurityConfiguration, ResourceType : AWS::EMR::SecurityConfiguration, StatusReason : Resource creation Initiated}, {LogicalResourceId : LogBucket, ResourceType : AWS::S3::Bucket, StatusReason : Resource creation Initiated}, {LogicalResourceId : ServiceRole, ResourceType : AWS::IAM::Role, StatusReason : Resource creation Initiated}, {LogicalResourceId : SC-117579184231-pp-dcyb4nvdu4pgq, ResourceType : AWS::CloudFormation::Stack, StatusReason : User Initiated}]
ssvegaraju commented 1 year ago

Hi, thanks for bringing up this issue! We've added this to our backlog, and will post here if there are any updates!

Thanks, Sukeerth