feat: Ollama Support - Githubissues

peperunas commented 1 month ago

Feature Description

It would be great to support Ollama in addition to the OpenAI API.

Ollama API: https://github.com/ollama/ollama/blob/main/docs/api.md

Use Case

Private LLMs

Additional context

Ollama supports part of the OpenAI endpoints (https://github.com/ollama/ollama/blob/main/docs/openai.md). I tried to set the environment variables accordingly but Morphic does not produce any result.

Ollama logs:

On Ollama's side I see a bunch of 400s:

May 31 21:58:12 architect ollama[2285650]: [GIN] 2024/05/31 - 21:58:12 | 400 |        63.5µs |       127.0.0.1 | POST     "/v1/chat/completions"
May 31 21:58:12 architect ollama[2285650]: [GIN] 2024/05/31 - 21:58:12 | 400 |     162.902µs |       127.0.0.1 | POST     "/v1/chat/completions"
May 31 21:58:28 architect ollama[2285650]: [GIN] 2024/05/31 - 21:58:28 | 400 |       83.97µs |       127.0.0.1 | POST     "/v1/chat/completions"
May 31 21:58:28 architect ollama[2285650]: [GIN] 2024/05/31 - 21:58:28 | 400 |      85.871µs |       127.0.0.1 | POST     "/v1/chat/completions"

Morphic logs

On Morphic side this is the error:

n [AI_APICallError]: json: cannot unmarshal array into Go struct field Message.messages.content of type string
    at O.errorSchema.x.z.object.error.x.z.object.message (/var/task/.next/server/chunks/430.js:66:4263)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async v (/var/task/.next/server/chunks/430.js:66:2056)
    at async S.doGenerate (/var/task/.next/server/chunks/430.js:66:7009)
    at async g (/var/task/.next/server/chunks/111.js:85:651569)
    at async E (/var/task/.next/server/chunks/111.js:85:653181)
    at async w (/var/task/.next/server/chunks/766.js:1:32191)
    at async /var/task/.next/server/chunks/766.js:73:1786 {
  url: 'https://ollamahost.com/v1/chat/completions',
  requestBodyValues: {
    model: 'pino:latest',
    logit_bias: undefined,
    user: undefined,
    max_tokens: undefined,
    temperature: undefined,
    top_p: undefined,
    frequency_penalty: undefined,
    presence_penalty: undefined,
    seed: undefined,
    messages: [ [Object], [Object] ],
    tool_choice: { type: 'function', function: [Object] },
    tools: [ [Object] ]
  },
  statusCode: 400,
  responseBody: '{"error":{"message":"json: cannot unmarshal array into Go struct field Message.messages.content of type string","type":"invalid_request_error","param":null,"code":null}}',
  cause: undefined,
  isRetryable: false,
  data: {
    error: {
      message: 'json: cannot unmarshal array into Go struct field Message.messages.content of type string',
      type: 'invalid_request_error',
      param: null,
      code: null
    }
  }
}

It seems that Morphic is sending invalid requests to Ollama?

I started working on a PR to add Ollama's support (#190 ). I'm currently testing it.