Open Manouchehri opened 9 months ago
Lower priority since we aren't using Amazon Bedrock as much after discovering that our AWS Activate credits don't apply to Anthropic or Cohere models on Bedrock. =P
Small note: I'm not even 100% sure how to use Cloudflare AI Gateway with Bedrock (ignoring LiteLLM), so don't worry about this ticket at all in the meantime. Not a high priority at all imo.
This is doable using boto3 hooks. I helped Cloudflare figure this one out, they will eventually be documenting it at https://developers.cloudflare.com/ai-gateway/providers/bedrock/
In the meantime, gonna save my notes here just in case.
https://stackoverflow.com/a/66097613/2079814
event_system.register_first('after-sign.*.*', _add_header)
or:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html#before-send
before-send
looks like it'd work too.
Unlike my example, you should only only boto3 requests going to bedrock invoke and invoke stream, since hooking others is just gonna break things if eventually the same boto3 client is used to do other API calls in the future.
JavaScript/TypeScript solution here: https://gist.github.com/Manouchehri/d1fa65b6d842a76d836cb6b3f9e1bbff
The Feature
The docs aren't added yet, but Cloudflare has added AI Gateway support for Amazon Bedrock.
https://developers.cloudflare.com/ai-gateway/providers/
The example request looks like:
Motivation, pitch
Free caching :D
Twitter / LinkedIn details
https://www.linkedin.com/in/davidmanouchehri/