Support Azure OpenAI

[tw4452852]

I've managed to use Azure OpenAI with the following minor change:

diff --git a/lib/chat_models/chat_open_ai.ex b/lib/chat_models/chat_open_ai.ex
index 5dd610e..06b4d0c 100644
--- a/lib/chat_models/chat_open_ai.ex
+++ b/lib/chat_models/chat_open_ai.ex
@@ -57,13 +57,13 @@ defmodule LangChain.ChatModels.ChatOpenAI do
           {:ok, Message.t() | MessageDelta.t() | [Message.t() | MessageDelta.t()]}
           | {:error, String.t()}

-  @create_fields [:model, :temperature, :frequency_penalty, :n, :stream, :receive_timeout]
+  @create_fields [:endpoint, :model, :temperature, :frequency_penalty, :n, :stream, :receive_timeout]
   @required_fields [:model]

   @spec get_org_id() :: String.t() | nil
@@ -220,7 +220,7 @@ defmodule LangChain.ChatModels.ChatOpenAI do
       Req.new(
         url: openai.endpoint,
         json: for_api(openai, messages, functions),
-        auth: {:bearer, get_api_key()},
+        headers: %{"api-key" => get_api_key()},
         receive_timeout: openai.receive_timeout
       )

Hope to make a general base to adapt this change to support more chat models.

[cigrainger]

This would be great. I'd love to support Claude as well. It seems that perhaps chat_model could be a behaviour. @brainlid would you be willing to entertain a PR making that change? That would make it easy enough to support Bumblebee a la #26.

[brainlid]

@cigrainger I'm just starting to take on Mistral via Bumblebee. Since I haven't done anything with a LLM and Bumblebee, it will help me think through how it might work and what other changes make sense.

[brainlid]

I hadn't seen Claude before. So many!

[chrisbarker]

@brainlid and everyone - I have an implementation for Azure OpenAI that has these and a few other changes to handle streaming responses that contain chunks of json, etc. I thought it might be worth breaking out as it's own chat_model as python langchain has?

I haven't PR'd it yet because I didn't consider it finished but I could push it in the next day or so if it is useful?

(I also have a sagemaker endpoint that is very llama2 specific if that is also helpful?)

[brainlid]

Hi @chrisbarker! If the implementation is identical to OpenAI, the all that needs to change is the endpoint. Since we're not doing inheritance, a function that returns the ChatOpenAI struct configured for Azure might be interesting to try.

It would change that developer API from something like ChatOpenAI.new(%{...}) to ChatModels.azure_open_ai(%{...}). :thinking:

I'd rather not duplicate the entire module for a namespace change.

[chrisbarker]

@brainlid there is a bit more that has changed, from #60

Based on adapting openai_chat_model to work with azure openai where the content moderation and chat completion api can result in json that is larger than the chunks being returned; requiring a little additional processing before parsing.

I've pushed a temp branch in my fork so it is easy to compare, see here (highlighted biggest change) https://github.com/chrisbarker/langchain/compare/azure_openai_chat_model...chrisbarker:langchain:tmp_for_compare_openai_to_azure_openai#diff-98396b383d660bb5274db90a5fd13b1526d5d6b120a2d8ece0e8baf06e55b85eL292-R331

I like the idea of struct but my gut reaction is it won't quite get us there but I'm open to suggestions and still/always learning

[brainlid]

@chrisbarker Thanks for the diff!

On this line:

It says:

JSON strings can be split over multiple responses we need to buffer the response until we have valid JSON strings to decode. JSON strings are delimited by two new lines.

This is already taking into account. This applies to a function_call which we don't attempt to decode until all the deltas for the message are received, combined and we have the "complete" message.

What we do is combine the JSON string response along the way. Then when the message is complete we can JASON decode it.

[brainlid]

Looking through the changes, it appears there are 2 issues being addresses:

• auth is handled differently
• something around JSON handling may be different

Is this correct?

[chrisbarker]

@brainlid - yes those are the issues.

The JSON issue occurs when handling deltas while streaming a text response from the LLM. A single delta can be split over more than one response chunk because the size of the JSON can be larger due to additional information, mainly about the content moderation, that is included when using the azure api.

[brainlid]

@brainlid - yes those are the issues.

The JSON issue occurs when handling deltas while streaming a text response from the LLM. A single delta can be split over more than one response chunk because the size of the JSON can be larger due to additional information, mainly about the content moderation, that is included when using the azure api.

Interesting. Can you paste in examples of what's received? I haven't seen the content moderation before either.

[chrisbarker]

@brainlid - here is an example where I am inspecting raw_data of each response chunk received. As you can see we can get multiple deltas in a single response and occasionally they don't align neatly across the chunks.

inspect

        {:data, raw_data}, response ->
          raw_data |> IO.inspect(label: "DATA:- ", printable_limit: :infinity)

output

DATA:- : "data: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\",\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" if\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" there\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" are\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" specific\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" details\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" about\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" the\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" new\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" developments\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" or\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" the\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" potential\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" value\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" they\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" could\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" bring\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" to\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" The\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" third\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"s"
DATA:- : "afe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" company\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\"2\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\",\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" be\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\n"
DATA:- : "data: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" sure\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" to\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" include\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" those\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" as\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" well\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\ndata: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\".\"},\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}]}\n\n"
DATA:- : "data: {\"id\":\"chatcmpl-8arLKTxmyC79hRBZdCAHBJS0hREPa\",\"object\":\"chat.completion.chunk\",\"created\":1703795550,\"model\":\"gpt-4\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"delta\":{},\"content_filter_results\":{}}]}\n\n"
DATA:- : "data: [DONE]\n\n"

[chrisbarker]

@brainlid nudge on the above in case you didn't see it over the holidays.

[brainlid]

Hi @chrisbarker! Sorry for the delay. No, I didn't find time during the holidays to dig in.

Since the module and model are basically OpenAI's ChatGPT, I'm wondering if it would be cleaner to just merge in support for the differences?

It sounds like it might be this:

• different endpoint
• different way of expressing the model
• need to handle the deltas when split across data chunks

I may be way off base with this, but wondering if you think this idea may work? Partly because I know there are many other services that support the OpenAI API model while being implemented by other servers.

What I'm hoping to balance:

• try to make it generic or override-able enough to not create multiple nearly-identical implementations
• balance that with not overly complicating the module

A separate question/request is to have some tests that explicitly demonstrate and handle the data example you included. I scanned for them but didn't see it and may have just missed it.

What are your thoughts?

[michalwarda]

@brainlid I'll add my 2 cents here that today when using chat I've stumbled upon the same issue using the openai endpoints (the data message spread across 2 server side events).

Can't seem to find any tests or even implementation in official openai client for this either. Seems everyone in every client is just doing a simple JSON parse on each event.

Can it be that idk. Mint has a "max_body" or something as a limit for the size of 1 chunk of stream it proceses?

[brainlid]

@michalwarda interesting idea. I haven't seen that myself yet. But I also haven't personally seen the content_filter_results portion of the message either.

Is that new? Is that an option?

[brainlid]

@chrisbarker From the InstructorEx discussion, I learned about Jaxon, a library that lets us convert partial JSON text.

I'd like to get some tests setup with the split chunks example and see if this can help us process it.

[michalwarda]

I've just got this error again. I'm using gpt-3.5-turbo directly through OpenAI api.

[error] Received invalid JSON: %Jason.DecodeError{position: 177, token: nil, data: "{\"id\":\"chatcmpl-8lZf0buihhFh5SCZrrKbnkiC7RFhu\",\"object\":\"chat.completion.chunk\",\"created\":1706349186,\"model\":\"gpt-3.5-turbo-1106\",\"system_fingerprint\":\"fp_b57c83dd65\",\"choices\":"}
[error] Received invalid JSON: %Jason.DecodeError{position: 74, token: nil, data: "[{\"index\":0,\"delta\":{\"content\":\".\"},\"logprobs\":null,\"finish_reason\":null}]}"}
[error] Received invalid JSON: %Jason.DecodeError{position: 68, token: nil, data: "{\"id\":\"chatcmpl-8lZf0buihhFh5SCZrrKbnkiC7RFhu\",\"object\":\"chat.comple"}
[error] Received invalid JSON: %Jason.DecodeError{position: 0, token: nil, data: "tion.chunk\",\"created\":1706349186,\"model\":\"gpt-3.5-turbo-1106\",\"system_fingerprint\":\"fp_b57c83dd65\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" a\"},\"logprobs\":null,\"finish_reason\":null}]}"}

I'll try to write down some tests today to make it easier to implement a solution.

[adampash]

Fwiw, I've also seen this behavior from OpenAI's chat completion APIs. This is prob not the greatest solution, but here's how I was dealing with it in elsewhere:

defmodule OpenAI.StreamAccumulator do
  @moduledoc """
  A module for accumulating and assembling streaming responses from OpenAI's API.

  This is required b/c occasionally the data chunks don't come back fully assembled in the event stream,
  so the json parsing can sometimes fail b/c of an incomplete message.
  """

  @default_state %{
    accumulator: "",
    data_events: [],
    callback: nil,
  }

  def start_link(callback) do
    Agent.start_link(fn -> Map.put(@default_state, :callback, callback) end)
  end

  def reset(pid) do
    Agent.update(pid, fn _ -> @default_state end)
  end

  def add_data(pid, data) do
    Agent.update(pid, fn state ->
      new_acc = state.accumulator <> data
      updated_acc =
        new_acc
        # trimming to ignore empty strings
        |> String.split("data:", trim: true)
        |> Enum.map(fn str ->
          str
          |> String.trim()
          |> decode_body(state.callback)
        end)
        |> Enum.filter(fn d -> d != :ok end)
        |> Enum.join("")
      %{
        state |
        accumulator: updated_acc,
      }
    end)
  end

  def get_state(pid) do
    Agent.get(pid, fn state -> state end)
  end

  def stop(pid) do
    Agent.stop(pid)
  end

  defp decode_body("", _), do: :ok
  defp decode_body("[DONE]", _), do: :ok

  defp decode_body(json, cb) do
    case Jason.decode(json) do
      {:ok, json} ->
        cb.(json)
        :ok

      {:error, reason} ->
        IO.inspect(reason, label: "Something went wrong parsing the response JSON")
        "data:" <> json
    end
  end

end

[brainlid]

The JSON parsing issue is being addressed separately in Issue #79

[brainlid]

I released v0.1.9 which attempts to handle the split delta messages. Please give this another try!

@michalwarda @adampash @chrisbarker

[raulchedrese]

I've been using langchain with Azure and it's working great aside from the authentication issue. I had to add a custom header to support Azure API keys https://github.com/raulchedrese/langchain/commit/f4cde700d79ed0a06817b71f84399a1ff74bff53.

[brainlid]

Thanks for the report @raulchedrese! I'd love help maintaining the Azure side as I don't use that. I don't want to be paying for X number of LLM services just to enable support for it.

That looks like an easy fix for Azure. I see you modified the OpenAI implementation. OpenAI requires an Authorization bearer header.

A number of services have implemented API compatibility with OpenAI's version, but the auth part might be different. Thinking about how that part could be customizable... :thinking:

If you have any thoughts on how to design that I'd love to hear!

[raulchedrese]

My commit adds the api-key header but doesn't remove the Authorization header so it should work with both Azure and OpenAI. If you're ok with the idea of always sending both that change would avoid needing to add any conditional logic.

I'd be happy to help out with Azure support. I'm working with it closely at the moment.

[brainlid]

@raulchedrese, thanks for identifying the api-key change needed. I've merged #93 which adds this change.

Please give it a try and let me know that it's working correctly for you. Thanks!

[brainlid]

I believe Azure OpenAI is supported now. Please reopen if that's not the case and provide some details.