search

aws-samples / amazon-bedrock-audio-summarizer

An automated way to transcribe and summarize media files using Amazon S3, AWS Lambda, Amazon Transcribe, and Amazon Bedrock.
https://www.allthingsdistributed.com/2024/05/hacking-our-way-to-better-team-meetings.html
MIT No Attribution
95 stars 20 forks source link

CDKToolkit failed to deploy resulting in CREATE_FAILED from permissions error. #8

Open jacido opened 1 month ago

jacido commented 1 month ago

Repo Link: https://github.com/aws-samples/amazon-bedrock-audio-summarizer/tree/main?tab=readme-ov-file

Below Quickstart in the Prerequisites section. There is not a mention of additional IAM roles required. Specifically there are 4 IAM permissions mentioned and during deployment of the solution there were several IAM permission errors:

Where we encountered permission errors:

  1. Roles

    • CloudFormationExecutionRole
    • ImagePublishingRole
    • FilePublishingRole

  2. Resources

    • ssm:DeleteParameter
    • ecr:CreateRepository

Expected behavior: CDKtoolkit stack deployment would be successful in Step 4 allowing to proceed to Step 5.

Behavior that occurred during the bootstrap:

Environment aws://8417XXXXXXXX/ap-south-1 failed bootstrapping: Error [ValidationError]: Stack:arn:aws:cloudformation:ap-south-1:8417XXXXXXXX:stack/CDKToolkit/37fcd710-0ef8-11ef-9f07-0244757fd391 is in DELETE_FAILED state and can not be updated. at Request.extractError (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:46692) at Request.callListeners (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:91600) at Request.emit (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:91048) at Request.emit (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:199651) at Request.transition (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:193203) at AcceptorStateMachine.runTo (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:158075) at /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:158405 at Request. (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:193495) at Request. (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:199726) at Request.callListeners (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:91768) { code: 'ValidationError', time: 2024-05-10T18:10:33.045Z, requestId: '08285ca9-60bf-4c0e-97bc-9909b1daef3f', statusCode: 400, retryable: false, retryDelay: 378.37039647842306 }

Additional error encountered: Potential Signal Echo after deleting the CREATE_FAILED CDKToolkit stack and deleting the S3 bucket. Upon initating the new bootstrap, encoutered Signal Echo.

(venv) (base) nish@Nishants-MacBook-Air amazon-bedrock-audio-summarizer % cdk bootstrap ⏳ Bootstrapping environment aws://8417XXXXXXXX/ap-south-1... Trusted accounts for deployment: (none) Trusted accounts for lookup: (none) Using default execution policy of 'arn:aws:iam::aws:policy/AdministratorAccess'. Pass '--cloudformation-execution-policies' to customize. CDKToolkit: creating CloudFormation changeset... 12:48:15 AM | CREATE_FAILED | AWS::S3::Bucket | StagingBucket Resource handler returned message: "cdk-hnb659fds-assets-8417XXXXXXXX-ap-south-1 already exists (Service: S3, Status Cod e: 0, Request ID: null)" (RequestToken: e2d46156-eXXX-2326-2a54-3b2aXXXXX, HandlerErrorCode: AlreadyExists)

❌ Environment aws://8417XXXXXXXX/ap-south-1 failed bootstrapping: Error: The stack named CDKToolkit failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "cdk-hnbXXXXX-assets-8417XXXXXXXX-ap-south-1 already exists (Service: S3, Status Code: 0, Request ID: null)" (RequestToken: e2dXXXXX-e598-2326-2a54-3b2a107XXXXX, HandlerErrorCode: AlreadyExists) at FullCloudFormationDeployment.monitorDeployment (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:440:10568) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:445:XXXX at async Promise.all (index 0) at async CdkToolkit.bootstrap (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:445:2174) at async exec4 (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:498:54396)

Next steps to resolve:

  1. Add missing IAM permissions:
    • ssm:DeleteParameter
    • ecr:CreateRepository (Added AmazonEC2ContainerRegistryReadOnly) according to documentation.

Link: https://docs.aws.amazon.com/AmazonECR/latest/userguide/security_iam_id-based-policy-examples.html#security_iam_id-based-policy-examples-console

  1. Grant Admin access (In lieu of AWS::IAM::Role's below missing)
    • CloudFormationExecutionRole
    • ImagePublishingRole
    • FilePublishingRole

According to documentation, CDK are assumed by the AWS CDK Toolkit to publish assets into an environment: that is, to write to the S3 bucket and the ECR repository, respectively.

  1. Delete bootstrap stack and S3 bucket then redeploy CDK stack.

Result: CDK solution deployed successfully upon re-attempt.

Fix request: Include roles and additional permissions needed in the Prerequisites section.

jacido commented 1 month ago

We enabled full admin for the account. The customer is requesting what permissions he'll need for least privilege.

Used github rep example for Code source eventbridge-bedrock-inference. Customer uploaded an audio file manually 30 minutes ago to the S3 bucket from Step 5a of the repo which should have created a transcription folder which should process the transcription to create the summary to place into the process folder. Following the architecture path , after transcribe the from Amazon EventBridge it did not cross to the event-bridge-bedrock-inference. Eventbridge function was not triggered by the update.

Lambda name: eventbridge-bedrock-inference

erikhopf commented 3 weeks ago

@jacido - were you able to make any progress on this with the customer?