No response from .ainvoke sometimes

[NicolasHoferer]

We are experiencing issues when using langchain-aws to connect to the bedrock-runtime service in AWS. Sometimes we don't get any response. We could not find any helpful information in our logs, not even with DEBUG log level. The issue is critical because our application requires minimal latency, and even a 3-second timeout can impact performance.

It does not happen everytime only sometimes and until now only when we run the application in the Azure cloud.

Anyone has an idea or similar issues?

Here is the code we use to setup the ChatBedrock using boto3.

def _get_boto3_client(region=None):
    region_name = config().aws_region
    session = boto3.Session()
    boto3_config = Config(connect_timeout=3, read_timeout=30)
    return session.client(
        service_name='bedrock-runtime',
        region_name=region_name,
        aws_access_key_id=config().aws_access_key_id,
        aws_secret_access_key=config().aws_secret_access_key,
        config=boto3_config
    )

bedrock_client = _get_boto3_client(region)
return ChatBedrock(
    client=bedrock_client,
    model_id=model_id,
    model_kwargs={"temperature": temperature, "max_tokens": 4096},
    callbacks=callbacks,
)

[3coins]

@NicolasHoferer Thanks for reporting this issue. Acknowledge that this is a concern, but would need more data to reproduce this and verify if this is an issue with the LangChain implementation or an underlying issue with the Bedrock service.

It does not happen everytime only sometimes and until now only when we run the application in the Azure cloud.

One thing that could help here is turning on the CloudWatch logs in AWS for Bedrock services, so that you can see if the request completed on the Bedrock side. Here is the documentation to turn on monitoring for Bedrock. https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html