search

sgomez / ollama-ai-provider

Vercel AI Provider for running LLMs locally using Ollama
https://www.npmjs.com/package/ollama-ai-provider
Other
167 stars 21 forks source link

Parsing error with `streamText`: `[AI_JSONParseError: JSON parsing failed` #7

Closed KastanDay closed 4 months ago

KastanDay commented 5 months ago

Using streamText with Ollama provider yields AI_JSONParseError.

It seems like everything's working, except it tries to JSON.parse() json snippit before they're fully read into the buffer.

Any guidance or help would be appreciated. Thanks for creating this package.

Code to reproduce (taken from your examples)

import { streamText } from 'ai'
import { createOllama } from 'ollama-ai-provider'

const result = await streamText({
    maxTokens: 1024,
    messages: [
      {
        content: 'Hello!',
        role: 'user',
      },
    ],
    model: model,
    system: 'You are a helpful chatbot.',
  })

  console.log("after starting streamtext. Result:", result)

  for await (const textPart of result.textStream) {
    console.log('OLLAMA TEXT PART:', textPart)
  }
  return result

Full error: 

 [AI_JSONParseError: JSON parsing failed: Text: {"model":"llama3:8b","created_at":"2024-06-24T23:49:13.635502488Z","message":{"role":"assistant","content":"Don"},"done":false}
{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.644880728Z","message":{"role":"assistant","content":"'t"},"done":false}
{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.653954767Z","message":{"role":"assistant","content":" worry"},"done":false}
{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.663009567Z","message":{"role":"assistant","content":","},"done":false}
{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.672704366Z","message":{"role":"assistant","content":" I"},"done":false}
{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.681958526Z","message":{"role":"assistant","content":"'m"},"done":false}
{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.691041205Z","message":{"role":"assistant","content":" here"},"done":false}
{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.700136604Z","message":{"role":"assistant","content":" to"},"done":false}
{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.709239924Z","message":{"role":"assistant","content":" assist"},"done":false}
{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.723447603Z","message":{"role":"assistant","content":" you"},"done":false}
{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.750979481Z","message":{.
Error message: Unexpected token { in JSON at position 128] {
  name: 'AI_JSONParseError',
  cause: [SyntaxError: Unexpected token { in JSON at position 128],
  text: '{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.635502488Z","message":{"role":"assistant","content":"Don"},"done":false}\n{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.644880728Z","message":{"role":"assistant","content":"\'t"},"done":false}\n{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.653954767Z","message":{"role":"assistant","content":" worry"},"done":false}\n{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.663009567Z","message":{"role":"assistant","content":","},"done":false}\n{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.672704366Z","message":{"role":"assistant","content":" I"},"done":false}\n{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.681958526Z","message":{"role":"assistant","content":"\'m"},"done":false}\n{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.691041205Z","message":{"role":"assistant","content":" here"},"done":false}\n{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.700136604Z","message":{"role":"assistant","content":" to"},"done":false}\n{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.709239924Z","message":{"role":"assistant","content":" assist"},"done":false}\n{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.723447603Z","message":{"role":"assistant","content":" you"},"done":false}\n{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.750979481Z","message":{'
}

Here's the StreamTextResult. Is it a problem that the reader is undefined? reader: undefined,

StreamTextResult {
  originalStream: ReadableStream {
  [Symbol(kType)]: 'ReadableStream',
  [Symbol(kState)]: <ref *1> {
  disturbed: false,
  reader: undefined,
  state: 'readable',
  storedError: undefined,
  stream: undefined,
  transfer: {
  writable: undefined,
  port1: undefined,
  port2: undefined,
  promise: undefined
},
  controller: ReadableStreamDefaultController {
  [Symbol(kType)]: 'ReadableStreamDefaultController',
  [Symbol(kState)]: {
  cancelAlgorithm: [Function: nonOpCancel],
  closeRequested: false,
  highWaterMark: 1,
  pullAgain: false,
  pullAlgorithm: [Function: nonOpPull],
  pulling: false,
  queue: [],
  queueTotalSize: 0,
  started: false,
  sizeAlgorithm: [Function],
  stream: ReadableStream {
  [Symbol(kType)]: 'ReadableStream',
  [Symbol(kState)]: [Circular *1],
  [Symbol(nodejs.webstream.isClosedPromise)]: {
  promise: Promise {
  [Symbol(async_id_symbol)]: 819253,
  [Symbol(trigger_async_id_symbol)]: 813769,
  [Symbol(kResourceStore)]: undefined,
  [Symbol(kResourceStore)]: undefined,
  [Symbol(kResourceStore)]: undefined
},
  resolve: [Function],
  reject: [Function]
},
  [Symbol(nodejs.webstream.controllerErrorFunction)]: [Function: bound error]
}
}
}
},
  [Symbol(nodejs.webstream.isClosedPromise)]: {
  promise: Promise {
  [Symbol(async_id_symbol)]: 819253,
  [Symbol(trigger_async_id_symbol)]: 813769,
  [Symbol(kResourceStore)]: undefined,
  [Symbol(kResourceStore)]: undefined,
  [Symbol(kResourceStore)]: undefined
},
  resolve: [Function],
  reject: [Function]
},
  [Symbol(nodejs.webstream.controllerErrorFunction)]: [Function: bound error]
},
  warnings: [],
  rawResponse: {
  headers: {
  alt-svc: 'h3=":443"; ma=86400',
  cf-cache-status: 'DYNAMIC',
  cf-ray: '8990cf0e9dd2c551-SEA',
  connection: 'keep-alive',
  content-type: 'application/x-ndjson',
  date: 'Tue, 25 Jun 2024 00:23:30 GMT',
  nel: '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}',
  report-to: '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v4?s=Bl6yODzpXFHa%2BBQ5SOg8MhYQnRBeHX39%2F%2Bt9L%2B1pQP3s8fvJoOxP%2BR2%2BzpTY2xJiQpefPoo%2BVq4oqyZXa7HBqc7FwPFc0HlJzPWbwmudhT2sxK0CE2QGCzPZgjFeydCaaQ%3D%3D"}],"group":"cf-nel","max_age":604800}',
  server: 'cloudflare',
  transfer-encoding: 'chunked'
}
}

Environment

idea2547 commented 5 months ago

If you doing this on localhost Make sure your both-side on client and ollama server is same http: or both should be https:

sgomez commented 5 months ago

I cannot reproduce the error @KastanDay . Can u give me the version of ollama and models are u using?

ollama -v
ollama list llama3

Also if u can try with other models. JSON generation had some problems in ollama, specially with some models. I knew that some bugs was fixed in the last ollama versions and I hope to have time next week to launch a new release of the module.

KastanDay commented 5 months ago

Hi sgomez thanks for your support. I've tried all your debugging steps below, still facing the same issue.

I'd be happy to give you access to my Ollama server for a while to help you debug, just let me know and I'll send you the public URL.

Cloudflare tunnels

Could the problem be Cloudflare? I'm exposing Ollama via a Cloudflare Tunnel. Per the Ollama docs on Cloudflare Tunnels, I set the --http-host-header="localhost:11434" as instructed. But maybe the way cloudflare chunks up the snippets is causing problems? It does seem to be "choppier" (with small bursts and pauses during streaming) than when I'm on the same machine.

@idea2547 I'm connecting to my Ollama server like this: 

const ollama = createOllama({
  baseURL: 'https://ollama.<redacted>.ai/api',
})

Debugging info

I tested on Ollama v0.1.46 (latest release) and v0.1.43. I'm running their Docker image. 

$ ollama list llama3
NAME                            ID              SIZE    MODIFIED
llama3:8b                       365c0bd3c000    4.7 GB  44 hours ago
llama3:70b-instruct             786f3184aec0    39 GB   4 weeks ago
llama3:8b-instruct-fp16         ca471fe48cbc    16 GB   6 weeks ago
llama3:latest                   a6990ed6be41    4.7 GB  6 weeks ago
llama3:70b-instruct-q6_K        2c308ad057fe    57 GB   6 weeks ago
llama3:70b                      be39eb53a197    39 GB   6 weeks ago

Per your suggestion, I tried other models. Same exact behavior with phi3 and gemma:2b.

sgomez commented 5 months ago

Yep, according the logs:

`{"model":"llama3:8b","created_at":"2024-06-24T23:49:13.750979481Z","message":{.`

is incomplete, probably cloudflare is streaming it in a different way. I will analyze the way I do the join of the chunks and check the new utils of the stable version of vercel ai-tools. I guess I can replace my own json stream response handler.

KastanDay commented 5 months ago

That seems correct, thank you! Let me know if you need anything.

sgomez commented 5 months ago

I released the version 0.8.0 than uses a new JsonStreamResponseHandler. Please, @KastanDay can u update and check if that fix the issue?

This will also require the last version of vercel ai-sdk

KastanDay commented 5 months ago

Hi @sgomez, thank you. It appears to have improved!

Now it always errors in exactly the final message. The final message is a different shape than the rest, it includes summary metrics like total_duration, etc. Edit: after further testing, sometimes it still errors before the final message, same as above. Happens rather randomly, I think it's Cloudflare being weird :( 

# The last 3 messages (prettified)
{
  "model": "llama3:8b",
  "created_at": "2024-07-01T19:50:41.461849164Z",
  "message": {
    "role": "assistant",
    "content": ".sh"
  },
  "done": false
},
{
  "model": "llama3:8b",
  "created_at": "2024-07-01T19:50:41.471326642Z",
  "message": {
    "role": "assistant",
    "content": "`)."
  },
  "done": false
},
{
  "model": "llama3:8b",
  "created_at": "2024-07-01T19:50:41.48093712Z",
  "message": {
    "role": "assistant",
    "content": ""
  },
  "done_reason": "stop",
  "done": true,
  "total_duration": 7525335412,
  "load_duration": 1475800,
  "prompt_eval_count": 16,
  "prompt_eval_duration": 38475000,
  "eval_count": 772,
  "eval_duration": 7342451000
}

Updated dependencies

npm list ai  
└── ai@3.2.16

npm list ollama-ai-provider
└── ollama-ai-provider@0.9.1

Full error message

Right before ollama streaming...
 [AI_JSONParseError: JSON parsing failed: Text: {"model":"llama3:8b","created_at":"2024-07-01T19:49:07.129596822Z","message":{"role":"assistant","content":"Hello"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.132765022Z","message":{"role":"assistant","content":" there"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.14696934Z","message":{"role":"assistant","content":"!"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.161141978Z","message":{"role":"assistant","content":" It"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.175354656Z","message":{"role":"assistant","content":"'s"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.189034134Z","message":{"role":"assistant","content":" nice"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.202178812Z","message":{"role":"assistant","content":" to"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.21526525Z","message":{"role":"assistant","content":" meet"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.228400448Z","message":{"role":"assistant","content":" you"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.241509166Z","message":{"role":"assistant","content":"!"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.254661925Z","message":{"role":"assistant","content":" I"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.267805883Z","message":{"role":"assistant","content":"'m"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.280930561Z","message":{"role":"assistant","content":" here"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.294083639Z","message":{"role":"assistant","content":" to"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.307236877Z","message":{"role":"assistant","content":" help"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.320394635Z","message":{"role":"assistant","content":" with"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.333574393Z","message":{"role":"assistant","content":" any"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.346525871Z","message":{"role":"assistant","content":" questions"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.35950659Z","message":{"role":"assistant","content":" or"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.372517588Z","message":{"role":"assistant","content":" topics"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.385516426Z","message":{"role":"assistant","content":" you"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.394613585Z","message":{"role":"assistant","content":"'d"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.403703423Z","message":{"role":"assistant","content":" like"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.412812182Z","message":{"role":"assistant","content":" to"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.421887661Z","message":{"role":"assistant","content":" discuss"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.431009899Z","message":{"role":"assistant","content":"."},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.440152898Z","message":{"role":"assistant","content":" What"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.449286417Z","message":{"role":"assistant","content":"'s"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.458441016Z","message":{"role":"assistant","content":" on"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.467598614Z","message":{"role":"assistant","content":" your"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.476763813Z","message":{"role":"assistant","content":" mind"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.485901532Z","message":{"role":"assistant","content":" today"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.49505961Z","message":{"role":"assistant","content":"?"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.504220009Z","message":{"role":"assistant","content":" Do"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.513390688Z","message":{"role":"assistant","content":" you"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.522544327Z","message":{"role":"assistant","content":" have"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.531717245Z","message":{"role":"assistant","content":" a"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.540875164Z","message":{"role":"assistant","content":" specific"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.550055843Z","message":{"role":"assistant","content":" question"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.559263601Z","message":{"role":"assistant","content":","},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.56846796Z","message":{"role":"assistant","content":" or"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.577638959Z","message":{"role":"assistant","content":" would"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.586863157Z","message":{"role":"assistant","content":" you"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.596027116Z","message":{"role":"assistant","content":" like"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.605212155Z","message":{"role":"assistant","content":" some"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.614404713Z","message":{"role":"assistant","content":" recommendations"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.623591192Z","message":{"role":"assistant","content":" for"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.632772391Z","message":{"role":"assistant","content":" things"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.64195111Z","message":{"role":"assistant","content":" to"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.651143948Z","message":{"role":"assistant","content":" talk"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.660327467Z","message":{"role":"assistant","content":" about"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.669527986Z","message":{"role":"assistant","content":"?"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.678761904Z","message":{"role":"assistant","content":" I"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.687901863Z","message":{"role":"assistant","content":"'m"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.697099262Z","message":{"role":"assistant","content":" all"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.70626678Z","message":{"role":"assistant","content":" ears"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.715456819Z","message":{"role":"assistant","content":" ("},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.724641818Z","message":{"role":"assistant","content":"or"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.733815857Z","message":{"role":"assistant","content":" rather"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.743009655Z","message":{"role":"assistant","content":","},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.752196454Z","message":{"role":"assistant","content":" all"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.761359213Z","message":{"role":"assistant","content":" text"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.770545011Z","message":{"role":"assistant","content":")!"},"done":false}
{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.77983313Z","message":{"role":"assistant","content":""},"done_reason":"stop","done":true,"total_duration":853379439,"load_duration":1312920,"prompt_eval_count":24,"prompt_eval_duration":37965000,"eval_count":64,"eval_duration":661216000}
.
Error message: Unexpected token { in JSON at position 130] {
  name: 'AI_JSONParseError',
  cause: [SyntaxError: Unexpected token { in JSON at position 130],
  text: '{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.129596822Z","message":{"role":"assistant","content":"Hello"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.132765022Z","message":{"role":"assistant","content":" there"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.14696934Z","message":{"role":"assistant","content":"!"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.161141978Z","message":{"role":"assistant","content":" It"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.175354656Z","message":{"role":"assistant","content":"\'s"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.189034134Z","message":{"role":"assistant","content":" nice"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.202178812Z","message":{"role":"assistant","content":" to"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.21526525Z","message":{"role":"assistant","content":" meet"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.228400448Z","message":{"role":"assistant","content":" you"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.241509166Z","message":{"role":"assistant","content":"!"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.254661925Z","message":{"role":"assistant","content":" I"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.267805883Z","message":{"role":"assistant","content":"\'m"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.280930561Z","message":{"role":"assistant","content":" here"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.294083639Z","message":{"role":"assistant","content":" to"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.307236877Z","message":{"role":"assistant","content":" help"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.320394635Z","message":{"role":"assistant","content":" with"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.333574393Z","message":{"role":"assistant","content":" any"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.346525871Z","message":{"role":"assistant","content":" questions"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.35950659Z","message":{"role":"assistant","content":" or"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.372517588Z","message":{"role":"assistant","content":" topics"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.385516426Z","message":{"role":"assistant","content":" you"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.394613585Z","message":{"role":"assistant","content":"\'d"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.403703423Z","message":{"role":"assistant","content":" like"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.412812182Z","message":{"role":"assistant","content":" to"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.421887661Z","message":{"role":"assistant","content":" discuss"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.431009899Z","message":{"role":"assistant","content":"."},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.440152898Z","message":{"role":"assistant","content":" What"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.449286417Z","message":{"role":"assistant","content":"\'s"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.458441016Z","message":{"role":"assistant","content":" on"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.467598614Z","message":{"role":"assistant","content":" your"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.476763813Z","message":{"role":"assistant","content":" mind"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.485901532Z","message":{"role":"assistant","content":" today"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.49505961Z","message":{"role":"assistant","content":"?"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.504220009Z","message":{"role":"assistant","content":" Do"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.513390688Z","message":{"role":"assistant","content":" you"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.522544327Z","message":{"role":"assistant","content":" have"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.531717245Z","message":{"role":"assistant","content":" a"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.540875164Z","message":{"role":"assistant","content":" specific"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.550055843Z","message":{"role":"assistant","content":" question"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.559263601Z","message":{"role":"assistant","content":","},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.56846796Z","message":{"role":"assistant","content":" or"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.577638959Z","message":{"role":"assistant","content":" would"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.586863157Z","message":{"role":"assistant","content":" you"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.596027116Z","message":{"role":"assistant","content":" like"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.605212155Z","message":{"role":"assistant","content":" some"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.614404713Z","message":{"role":"assistant","content":" recommendations"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.623591192Z","message":{"role":"assistant","content":" for"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.632772391Z","message":{"role":"assistant","content":" things"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.64195111Z","message":{"role":"assistant","content":" to"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.651143948Z","message":{"role":"assistant","content":" talk"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.660327467Z","message":{"role":"assistant","content":" about"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.669527986Z","message":{"role":"assistant","content":"?"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.678761904Z","message":{"role":"assistant","content":" I"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.687901863Z","message":{"role":"assistant","content":"\'m"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.697099262Z","message":{"role":"assistant","content":" all"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.70626678Z","message":{"role":"assistant","content":" ears"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.715456819Z","message":{"role":"assistant","content":" ("},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.724641818Z","message":{"role":"assistant","content":"or"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.733815857Z","message":{"role":"assistant","content":" rather"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.743009655Z","message":{"role":"assistant","content":","},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.752196454Z","message":{"role":"assistant","content":" all"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.761359213Z","message":{"role":"assistant","content":" text"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.770545011Z","message":{"role":"assistant","content":")!"},"done":false}\n{"model":"llama3:8b","created_at":"2024-07-01T19:49:07.77983313Z","message":{"role":"assistant","content":""},"done_reason":"stop","done":true,"total_duration":853379439,"load_duration":1312920,"prompt_eval_count":24,"prompt_eval_duration":37965000,"eval_count":64,"eval_duration":661216000}\n'
}
joe-quinn commented 5 months ago

Ran into this same error today while using streamUI from 'ai/rsc'

Deps

"ai": "^3.2.16",
"ollama-ai-provider": "^0.9.1",

Error Message

 ⨯ AI_JSONParseError: JSON parsing failed: Text: {"model":"llama3","created_at":"2024-07-04T19:09:01.641444391Z","message":{"role":"assistant","content":" factors"},"done":false}
{"model":"llama3","created_at":"2024-07-04T19:09:01.666119003Z","message":{"role":"assistant","content":"."},"done":false}
.
Error message: Unexpected non-whitespace character after JSON at position 130
Cause: [SyntaxError: Unexpected non-whitespace character after JSON at position 130]
KastanDay commented 5 months ago

@joe-quinn do you use Cloudflare or Cloudflare tunnels anywhere in your stack? That’ll help clarify the root cause. Thanks.

joe-quinn commented 5 months ago

@joe-quinn do you use Cloudflare or Cloudflare tunnels anywhere in your stack? That’ll help clarify the root cause. Thanks.

@KastanDay I do not.

Also, it seems that this error is only occurring on local dev, on both http and https. So far, I have not been able to replicate the error on production (deployed on Vercel).

sgomez commented 4 months ago

Basically the error is some extra \n characters. I suspect than there are some reasons for that:

  1. Double \n\n from the stream.
  2. Wrong chunk division in the createJsonStreamResponseHandler method from vercel/ai-sdk.

I guess is the second one, because seems like the logic only expect one message per chunk. If there are many messages in one chunk, there are not spliced (and that is the reason to have those new line chars).

Anyway, the problem comes from a wrong data extraction from chunks.

I will create some tests to try to get the error and improve the chunk processing.

Thanks for your help.

sgomez commented 4 months ago

I released version 0.10.0 with a new line splitter stream before JSON stream processor. If u have very kind to confirm than it is working now. Thanks!

KastanDay commented 4 months ago

It finally worked!

Thanks for your tireless work on this @sgomez. Much appreciated.

Versions tested: 

"ai": "3.2.35",
"ollama-ai-provider": "0.10.0",

Reproducible code: 

import { streamText } from 'ai'
import { createOllama } from 'ollama-ai-provider'

export const ollamaStream = async () => {
  const ollama = createOllama({
    baseURL: 'https://<redacted>/api',
  })
  const result = await streamText({
    model: ollama('llama3.1:70b'),
    prompt: 'Invent a new holiday and describe its traditions.',
  })

  for await (const textPart of result.textStream) {
    console.log(textPart)
  }
}