[Feature Request] Cross Region Inference

statefb commented 2 weeks ago

Describe the solution you'd like

Some region does not support some models.

To enable inference for unsupported regions.

Only limited regions supports inference profile (currently us and eu only)
- we can check if the region supports by aws bedrock list-inference-profiles --profile <profile-name>
Due to the architecture to routing requests, latency happens (see doc).
- Add enableBedrockCrossRegionInference option to cdk.json with the default value false?
Even the value is set to true, if the bedrockRegion is not included in the supported region, model id without cross region inference should be chosen. Also warning log should be added.

Related PR: #531 (reverted) Related Issues: #508, #527

Are you willing to discuss the solution with us, decide on the approach, and assist with the implementation?

juan-abia commented 2 weeks ago

If I do cross region inference I get this error: Provider eu model does not support chat.

Any idea how to solve it?

axelpina commented 2 weeks ago

If I do cross region inference I get this error: Provider eu model does not support chat.

Any idea how to solve it?

There's a PR with a fix in progress but it hasn't been merged: https://github.com/aws-samples/bedrock-claude-chat/pull/536