Agents for Amazon Bedrock Runtime not returning source metadata in APIs

[tim-finnigan]

Original issue: https://github.com/boto/boto3/issues/4124 (ref: P131777621)

[sidatcd]

@tim-finnigan Any ETA on this?

[edu2105]

I was getting crazy searching for a parameter, flag or configuration I missed in order to show the source metadata. While the invoke_agent documentation shows that the metadata should come within the retrievedReferences object, in the documentation for the trace-events under OrchestrationTrace --> Observation the 'metadata' object it is not present https://docs.aws.amazon.com/bedrock/latest/userguide/trace-events.html

Thanks god I've found this open issue. Hope there is an update soon 🙏 and thanks for working on this.

[adoyon23]

I was able to resolve this by upgrading to boto3 1.34.118

[tim-finnigan]

Thanks for confirming @adoyon23 - I'll go ahead and close this issue. For anyone encountering this issue please update to the latest version of the AWS SDK. Here is the Boto3 CHANGELOG for reference, showing 1.34.118 as the latest version.

[github-actions[bot]]

This issue is now closed.

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[edu2105]

Hi @adoyon23 I've been trying with both 1.34.118 and the latest one 1.34.119 versions, but still cannot find the metadata object within retrievedReferences when calling the client.invoke_agent method. I can confirm that the metadata is showing when using the Knowledge Base playground but not when using the invoke_agent.

Did you have to enable or configure something else?

[sriram-aws]

Hi @tim-finnigan - I am still not able to find resolution for this issue. invoke_agent still doesnt have metadata in the output.

[tim-finnigan]

Hi @edu2105 @sriram-aws thanks for following up. After speaking with an engineer on the Bedrock team, I was told that they are aware of this issue and are planning a fix soon. Will keep this open to track for now.

[sriram-aws]

@tim-finnigan - Thanks for the update!

[motigors]

Hi,

I'm experiencing an issue with the bedrock-agent-runtime retrieve_and_generate service. When I invoke this service, the retrievedReferences (retrieve metadata) is not included in the response. This functionality worked until the middle of last week, but it has since stopped working with any version of boto3.

Please confirm if this is related to this issue, or should I open a new bug report it? If it is the same issue, do you have an estimated timeline for a fix?

python: 3.11 boto3: 1.34.122

Request:

input={
    'text': 'some question'
},
retrieveAndGenerateConfiguration={
    'type': 'KNOWLEDGE_BASE',
    'knowledgeBaseConfiguration': {
        'knowledgeBaseId': '1234',
        'modelArn': 'arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-haiku-20240307-v1:0',
        'retrievalConfiguration': {
            'vectorSearchConfiguration': {
                'numberOfResults': 3
            }
        },
        'generationConfiguration': {
            'inferenceConfig': {
                'textInferenceConfig': {
                    "temperature": 0.1,
                }
            }
        }
    },
}

Response:

{
    "ResponseMetadata": {
        "HTTPHeaders": {
            "connection": "keep-alive",
            "content-length": "1030",
            "content-type": "application/json",
            "date": "Sun, 09 Jun 2024 13:18:00 GMT",
            "x-amzn-requestid": "1234"
        },
        "HTTPStatusCode": 200,
        "RequestId": "1234",
        "RetryAttempts": 0
    },
    "citations": [
        {
            "generatedResponsePart": {
                "textResponsePart": {
                    "span": {
                        "end": 411,
                        "start": 0
                    },
                    "text": "generated response"
                }
            },
            "retrievedReferences": []
        }
    ],
    "output": {
        "text": "generated response"
    },
    "sessionId": "1234"
}

[tim-finnigan]

@motigors that appears to be a related issue. No timeline on addressing this but the Bedrock team informed me that they are working on it.

[motigors]

Thanks for the update

[ncagle]

@tim-finnigan I'm having a similar issue, but with the bedrock-agent-runtime retrieve method. It seems like it's related.

I'm glad to see this is being worked on! If there are any updates, it'd be great to hear. If my issue is unrelated, I can open a new issue if there isn't a relevant one already.

Here's some info about what I was doing in case it helps. I haven't been able use a RetrievalFilter in the vectorSearchConfiguration either. But I have confirmed in the bedrock console testing that the metadata exists and is working correctly for filtering.

Python 3.12 boto3: 1.34.42

response = bedrock_agent_runtime_client.retrieve(
    knowledgeBaseId="ABC1234567",
    # nextToken="",
    retrievalConfiguration={
        "vectorSearchConfiguration": {
            "numberOfResults": 6,
        }
    },
    retrievalQuery={
        "text": "some information"
    }
)

Response:

{
    'ResponseMetadata': {
        'HTTPHeaders': {
            'connection': 'keep-alive',
            'content-length': '16283',
            'content-type': 'application/json',
            'date': 'Thu, 20 Jun 2024 19:38:59 GMT',
            'x-amzn-requestid': 'totally-real-id'
        },
        'HTTPStatusCode': 200,
        'RequestId': 'totally-real-id',
        'RetryAttempts': 0
    },
    'retrievalResults': [
        {'content': {'text': 'relevant stuff and things'},
         'location': {'s3Location': {'uri': 's3://bucket/subdirectory/document_1.pdf'},
                      'type': 'S3'},
         'score': 0.7049113},
        {'content': {'text': 'relevant stuff and things'},
         'location': {'s3Location': {'uri': 's3://bucket/subdirectory/document_2.pdf'},
                      'type': 'S3'},
         'score': 0.704336},
        {'content': {'text': 'relevant stuff and things'},
         'location': {'s3Location': {'uri': 's3://bucket/subdirectory/document_3.pdf'},
                      'type': 'S3'},
         'score': 0.700758},
        {'content': {'text': 'relevant stuff and things'},
         'location': {'s3Location': {'uri': 's3://bucket/subdirectory/document_4.pdf'},
                      'type': 'S3'},
         'score': 0.70058656},
        {'content': {'text': 'relevant stuff and things'},
         'location': {'s3Location': {'uri': 's3://bucket/subdirectory/document_5.pdf'},
                      'type': 'S3'},
         'score': 0.7002422},
        {'content': {'text': 'relevant stuff and things'},
         'location': {'s3Location': {'uri': 's3://bucket/subdirectory/document_6.pdf'},
                      'type': 'S3'},
         'score': 0.6993249}
    ]
}