Connection for use sagemaker Endpoint

continuedev / continue

⏩ Continue is the leading open-source AI code assistant. You can connect any models and any context to build custom autocomplete and chat experiences inside VS Code and JetBrains

https://continue.dev/docs

Apache License 2.0

13.09k stars 887 forks source link

Connection for use sagemaker Endpoint #1130

Open theja0473 opened 2 months ago

theja0473 commented 2 months ago

Validations

[X] I believe this is a way to improve. I'll try to join the Continue Discord for questions
[X] I'm not able to find an open issue that requests the same enhancement

Problem

Currently Ollama or any self hosted services only allowed my suggestion lets connect to end point of Sagemaker and serve Copilot use case

Solution

No response

sestinj commented 2 months ago

Hi @theja0473 can you share more about what API format you expect for the Sagemaker endpoint?

mikhail-khodorovskiy commented 1 week ago

We are also interested in hosting the model in Sagemaker. The format of the API is:

https://runtime.sagemaker.us-west-2.amazonaws.com/endpoints/my-model-id/invocations

For boto, the documentation of how to invoke the endpoint: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker-runtime/client/invoke_endpoint.html

The credentials can be provided the same way as you have it implemented for bedrock: https://docs.continue.dev/reference/Model%20Providers/bedrock

tobiasbartsch commented 1 week ago

I am also very interested in hosting a model in Sagemaker. I think the main thing to add (as @mikhail-khodorovskiy pointed out) is credentials handling identical to how Bedrock is done.