Azure OpenAI: TypeError: Cannot read properties of undefined (reading 'delta')

[jeevanions]

I am using Azure OpenAI and trying to use this library for streaming the json object. Getting this error

error: TypeError: Cannot read properties of undefined (reading 'delta')
      at OpenAiHandler.process (/proj/backend/openai-partial-stream/packages/openai-partial-stream/dist/index.js:205:32)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async Entity.genParse (/proj/backend/openai-partial-stream/packages/openai-partial-stream/dist/index.js:270:22)
      at async /proj/backend/index.js:389:30

I tried to fix it like this but did not work

[st3w4r]

Hello thank you for reporting, do you have the actual raw content in msg to help debugging?

[thomashibbard]

For me it happens when enabling the pretty recently introduced stream_options.include_usage option. The final msg content looks like this:

{
   "id":"chatcmpl-<id here>",
   "object":"chat.completion.chunk",
   "created":1718756652,
   "model":"gpt-4-0613",
   "system_fingerprint":null,
   "choices":[

   ],
   "usage":{
      "prompt_tokens":308,
      "completion_tokens":68,
      "total_tokens":376
   }
}

[thomashibbard]

Are there plans to address this?

[st3w4r]

@thomashibbard thank you for the details, this should help with the investigation.

In the JSON payload you attached, there are no choices. Is that the complete payload?

If stream_options.include_usage triggers this, it could be due to some versions that need updating. I will have to investigate.

[thomashibbard]

@st3w4r Yeah exactly, there are no choices—I think—because the final received chunk is just token usage information, and so it doesn't have any of the choices/delta/etc included in the rest of the stream for composing responses.

[st3w4r]

I see, indeed if the last chunk have no choices we could just skip it. I’m wondering if there is something else that could help differentiate this chunk.

A curl command could confirm this behavior it.

[thomashibbard]

I think the last chunk still needs to be processed so the usage can be accessed. Maybe it could emit a done or usage event. I'm getting a little out of my depth here, but would something like this be possible

function process() {
  // Create an instance of the handler
  const openAiHandler = new OpenAiHandler(mode);
  // Process the stream. 
  const entityStream = openAiHandler.process(stream);
  // Create an entity with the schema to validate the data
  const entityPostcode = new Entity("postcodes", PostcodeSchema);
  // Parse the stream to an entity, using the schema to validate the data
  const postcodeEntityStream = entityPostcode.genParseArray(entityStream);

  postcodeEntityStream.on('usage', saveUsageToDb)

  return postcodeEntityStream
}