aws-samples / sample-connector-for-bedrock

Seamlessly invoke Amazon Bedrock or your custom models, enabling a smooth experience with AWS GenAI services.
https://aws-samples.github.io/sample-connector-for-bedrock/
MIT No Attribution
40 stars 12 forks source link

Add ollama backend support #4

Closed stevensu1977 closed 4 months ago

stevensu1977 commented 4 months ago

I add ollama backend support , after start ollama service , you can use "ollama:" prefix , etc. ollama:mistral:v0.3, ollama:llama3 , default ollama host is "http://localhost:11434"

curl http://localhost:8866/v1/chat/completions \
  -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer br-a8OTgDNsdF0vYsevxtoEeVPAyze7P" \
  -d '{
    "model": "ollama:mistral:v0.3",
    "messages": [
      {
        "role": "user",
        "content": "who are you"
      }
    ],
    "stream": true,
    "temperature": 0.1,
    "max_tokens": 4096
  }'

output:
{"choices":{"role":"assistant","content":" I am a model trained by Mistral AI. I was designed to assist with a wide range of tasks, answer questions, and engage in conversation on various topics. How can I help you today?"},"usage":{"completion_tokens":41,"prompt_tokens":7,"total_tokens":48}}

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.