[Bug]: langfuse has wrong token counts

Manouchehri commented 6 days ago

What happened?

curl -v "${OPENAI_API_BASE}/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gemini-1.5-pro-001",
    "max_tokens": 1000,
    "stream": true,
    "temperature": 0.0,
    "messages": [
      {
        "role": "user",
        "content": "Tell me the funniest joke you know."
      },
      {
        "role": "assistant",
        "content": "Why did the chicken cross the road?\nYou will not guess this one I bet\n"
      },
      {
        "role": "user",
        "content": "I do not know, why?"
      },
      {
        "role": "assistant",
        "content": "uhhhh\n\n\nhmmmm.....\nthinking....\n"
      },
      {
        "role": "user",
        "content": "\nI am waiting...\n\n...\n"
      }
    ],
    "stream_options": {
      "include_usage": true
    }
  }'

data: {"id":"chatcmpl-eae25cb3-1b59-43d9-b7bc-0fe2f3ea4e03","choices":[{"index":0,"delta":{"content":"To","role":"assistant"}}],"created":1719865807,"model":"gemini-1.5-pro-001","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-eae25cb3-1b59-43d9-b7bc-0fe2f3ea4e03","choices":[{"index":0,"delta":{"content":" get to the other....  Wait for it...  \n\n*Side of the"}}],"created":1719865808,"model":"gemini-1.5-pro-001","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-eae25cb3-1b59-43d9-b7bc-0fe2f3ea4e03","choices":[{"index":0,"delta":{"content":"—* \n\nTo prove to the opossum it **could** be done"}}],"created":1719865808,"model":"gemini-1.5-pro-001","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-eae25cb3-1b59-43d9-b7bc-0fe2f3ea4e03","choices":[{"index":0,"delta":{"content":"!  Opossums just play dead, you know!  \n\n😂  Get it?  \n\nLet me know if you want to hear another joke!"}}],"created":1719865809,"model":"gemini-1.5-pro-001","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-eae25cb3-1b59-43d9-b7bc-0fe2f3ea4e03","choices":[{"index":0,"delta":{"content":" 😄 \n"}}],"created":1719865809,"model":"gemini-1.5-pro-001","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-eae25cb3-1b59-43d9-b7bc-0fe2f3ea4e03","choices":[{"index":0,"delta":{"content":""}}],"created":1719865809,"model":"gemini-1.5-pro-001","object":"chat.completion.chunk","usage":{"prompt_tokens":50,"completion_tokens":68,"total_tokens":118}}

data: {"id":"chatcmpl-eae25cb3-1b59-43d9-b7bc-0fe2f3ea4e03","choices":[{"finish_reason":"stop","index":0,"delta":{}}],"created":1719865809,"model":"gemini-1.5-pro-001","object":"chat.completion.chunk"}

See this: {"prompt_tokens":50,"completion_tokens":68,"total_tokens":118}

Yet langfuse shows: