Helicone / helicone

🧊 Open source LLM-Observability Platform for Developers. One-line integration for monitoring, metrics, evals, agent tracing, prompt management, playground, etc. Supports OpenAI SDK, Vercel AI SDK, Anthropic SDK, LiteLLM, LLamaIndex, LangChain, and more. 🍓 YC W23
https://www.helicone.ai
Apache License 2.0
1.91k stars 211 forks source link

Helicone worker throws JSON parse error with regular OpenAI call #1549

Open thomasuebi opened 7 months ago

thomasuebi commented 7 months ago

The Helicone worker server throws the following error in completion:

POST /v1/chat/completions 500 Internal Server Error (104241.23ms) at ServiceWorkerGlobalScope.[kDispatchFetch] (/worker/helicone/worker/node_modules/@miniflare/core/src/standards/event.ts:385:13) at Object.handle (/worker/helicone/worker/node_modules/itty-router/index.mjs:1:614) at /worker/helicone/worker/src/routers/openaiProxyRouter.ts:20:12 at proxyForwarder (/worker/helicone/worker/src/lib/HeliconeProxyRequest/forwarder.ts:66:28) at getCachedResponse (/worker/helicone/worker/src/lib/cache/cacheFunctions.ts:73:52) at getMaxCachedResponses (/worker/helicone/worker/src/lib/cache/cacheFunctions.ts:95:20) at async Promise.all (index 0) at KVNamespace.get (/worker/helicone/worker/node_modules/@miniflare/kv/src/namespace.ts:209:12) at convertStoredToGetValue (/worker/helicone/worker/node_modules/@miniflare/kv/src/namespace.ts:128:19) at JSON.parse () SyntaxError: Unexpected number in JSON at position 1907

Completion call (using the official OpenAI node module:

 response = await openai.chat.completions.create({
            model: "gpt-3.5-turbo",
            messages: [
              {
                role: "system",
                content: `Summarize this item description with a maximum of 1-4 words, in plain german.
              Remove all prices (EUR, TL, etc.), remove all tokens that look like article-numbers, and quantities. 
              Remove any redundant information. Keep information that might be useful for customs declarations. Do not make up or add any information.
              The output is just the summarized text, without any explanation.`,
              },
              {
                role: "user",
                content: articleNumberDescription?.substring(0, 3000),
              },
            ],
            temperature: 0,
            max_tokens: 100,
            user: customerId,
          })

Where articleNumberDescription is a simple "Katheter für medizinische Zwecke"

OpenAI Config:

openai: {
    apiKey: keys["openai-key"],
    basePath: keys["openai-base-path"],
    baseURL: keys["openai-base-path"],
    defaultHeaders: {
      "Helicone-Auth": `Bearer ${keys["helicone-api-key"]}`,
      "Helicone-Cache-Enabled": "true",
      "Cache-Control": "max-age=2592000",
      "Helicone-Property-App": "data-normalization-service",
      "Helicone-Retry-Enabled": "true",
    },
  },

This is very strange behaviour. For me, that looks more like a bug than a user error. Any ideas?

colegottdank commented 7 months ago

Do you have the same error if you hit OpenAI directly?

Also, please set the messages to a variable and then console.log that variable? I want to see what the computed messages looks like