[BUG] AWS::Bedrock::KnowledgeBase resource creation via AgentKnowledgeBase construct fails due to the lack of permission for KnowledgeBase role depending on the timing.

Description of the issue

AWS::Bedrock::KnowledgeBase resource creation via AgentKnowledgeBase construct fails due to the lack of permission for KnowledgeBase role.

AgentWithKBandGuardrailsStack: deploying... [1/1]
AgentWithKBandGuardrailsStack: creating CloudFormation changeset...
1:14:37 AM | CREATE_FAILED        | AWS::Bedrock::KnowledgeBase
| BedrockDocsKnowledgeBaseC3AACD10
Resource handler returned message: "The knowledge base storage configuratio
n provided is invalid... Request failed: [security_exception] 403 Forbidden
(Service: BedrockAgent, Status Code: 400, Request ID: 638dfe04-aa0e-4ac3-b0
70-0024a3290790)" (RequestToken: f0d57e4d-ea42-9c15-87d9-7b7aa4f93860, Hand
lerErrorCode: InvalidRequest)

Reproductions steps

I used 03-agent-with-kb-and-guardrails in amazon-bedrock-samples repository as a sample.

./blueprints.sh deploy 03-agent-with-kb-and-guardrails

Investigation

From AgentKnowledgeBase construct, KnowledgeBase role (KB role) is created. https://github.com/awslabs/agents-for-amazon-bedrock-blueprints/blob/fdd45ca8d5198717ee9b84b7ececbecf69091d9e/lib/constructs/AgentKnowledgeBase.ts#L150-L169

For this role, some necessary permissions are granted by addToPolicy method.

https://github.com/awslabs/agents-for-amazon-bedrock-blueprints/blob/fdd45ca8d5198717ee9b84b7ececbecf69091d9e/lib/constructs/AgentKnowledgeBase.ts#L166 https://github.com/awslabs/agents-for-amazon-bedrock-blueprints/blob/fdd45ca8d5198717ee9b84b7ececbecf69091d9e/lib/constructs/AgentKnowledgeBase.ts#L193 https://github.com/awslabs/agents-for-amazon-bedrock-blueprints/blob/fdd45ca8d5198717ee9b84b7ececbecf69091d9e/lib/constructs/AgentKnowledgeBase.ts#L379

These statements generate AWS::IAM::Policy resource in CloudFormation template. However, AWS::Bedrock::KnowledgeBase resource does NOT depend on the AWS::IAM::Policy resource. As a result, CloudFormation will try to create AWS::Bedrock::KnowledgeBase and AWS::IAM::Policy for KB role in parallel. If the creation for AWS::Bedrock::KnowledgeBase resource is initiated before AWS::IAM::Policy resource for KB role depending on the timing, then CreateKnowldgeBase API will fail due to the lack of permission for KB role. * Note: If the creation for the AWS::IAM::Policy resource completes luckily before initiating creation for AWS::Bedrock::KnowledgeBase, the issue will not occur.

When the issue occurred, I confirmed InvokeModel API call by KB role failed with AccessDenied exception and then CreateKnowldgeBase API call by CloudFormation failed with ValidationException in CloudTrail logs.

    "eventTime": "2024-08-17T01:14:36Z",
    "eventSource": "bedrock.amazonaws.com",
    "eventName": "InvokeModel",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "bedrock.amazonaws.com",
    "userAgent": "bedrock.amazonaws.com",
    "errorCode": "AccessDenied",
    "errorMessage": "User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/AgentWithKBandGuardrailsS-BedrockDocsBedrockKBServi-OeBY4e6ms0pm/BedrockKnowledgeBaseCPSession-IGJDOHN98G is not authorized to perform: bedrock:InvokeModel on resource: arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-embed-text-v1 because no identity-based policy allows the bedrock:InvokeModel action",

    "eventTime": "2024-08-17T01:14:36Z",
    "eventSource": "bedrock.amazonaws.com",
    "eventName": "CreateKnowledgeBase",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "cloudformation.amazonaws.com",
    "userAgent": "cloudformation.amazonaws.com",
    "errorCode": "ValidationException",
    "requestParameters": {
        "clientToken": "f0d57e4d-ea42-9c15-87d9-7b7aa4f93860",
        "roleArn": "arn:aws:iam::xxxxxxxxxxxx:role/AgentWithKBandGuardrailsS-BedrockDocsBedrockKBServi-OeBY4e6ms0pm",
        "storageConfiguration": {
            "opensearchServerlessConfiguration": {
                "collectionArn": "arn:aws:aoss:us-east-1:xxxxxxxxxxxx:collection/rav4jj12yunwmh1k9x6a",
                "fieldMapping": {
                    "metadataField": "***",
                    "textField": "***",
                    "vectorField": "***"
                },
                "vectorIndexName": "***"
            },
            "type": "OPENSEARCH_SERVERLESS"
        },
        "name": "***",
        "knowledgeBaseConfiguration": {
            "type": "VECTOR",
            "vectorKnowledgeBaseConfiguration": {
                "embeddingModelArn": "arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-embed-text-v1"
            }
        },
        "tags": {
            "aws:cloudformation:stack-name": "AgentWithKBandGuardrailsStack",
            "aws:cloudformation:logical-id": "BedrockDocsKnowledgeBaseC3AACD10",
            "aws:cloudformation:stack-id": "arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/AgentWithKBandGuardrailsStack/cc437610-5c35-11ef-941e-0e42571afc69"
        }
    },
    "responseElements": {
        "message": "The knowledge base storage configuration provided is invalid... Request failed: [security_exception] 403 Forbidden"
    },

awslabs / agents-for-amazon-bedrock-blueprints