Compatible for Azure OpenAI Service

[xuhaoruins]

Describe the feature

hi all, I am using Azure OpenAI service. I have changed basePath to Azure OpenAI endpoint. But API didnt works. Does API compatible to Azure OpenAI like python version? any solutions here?

[transitive-bullshit]

The ChatGPT wrapper is not compatible with the Azure OpenAI service as far as I know.

We're using OpenAI's completions endpoint: https://platform.openai.com/docs/api-reference/completions

I am not sure whether or not Azure's hosted version is compatible with OpenAI's standalone version, but this package hasn't been tested with the Azure version.

[transitive-bullshit]

If you find a way to work with the Azure OpenAI service or can provide more info on the differences, I'll consider adding support.

Thanks

[xuhaoruins]

Hello Travis

The Python lib is compatitble to Azure OpenAI. I found a way to success get response from Azure endpoint. GitHub - xuhaoruins/azure-openai-quickstart-nodehttps://github.com/xuhaoruins/azure-openai-quickstart-node

Looks like the key should be in header and params need to be override.

Thanks, Hao

[joostgrunwald]

@transitive-bullshit @xuhaoruins You can fix this by not using completions but chat/completions instead:

POC: const url = https://{resource-name}.openai.azure.com/openai/deployments/{deployment-name}/chat/completions?api-version={date_preview_here}; const headers = { "Content-Type": "application/json", "api-key": ${this._apiKey} //Authorization: Bearer ${this._apiKey} };

You probably want to build a option for this and change this small piece of code based on that.

[namiwang]

If you find a way to work with the Azure OpenAI service or can provide more info on the differences, I'll consider adding support.

Thanks

FYI this is a proxy script to translate openai request to azure one

[bumu]

good ideas

[PvanHengel]

Morning - I have done a bunch of work to get it working with Azure. For starters the easiest way without making mods to the base library (which we need to do), is to override the fetch function in the constructor, you setup your apiBaseUrl like: https://{}.openai.azure.com/openai/deployments/{}'

then override the fetch function with a custom fetch function that takes the same args (url, opts), and returns fetch(newUrl, opts), you may need to adjust the API version etc..

const newUrl = `${url}?api-version=2023-03-15-preview`
opts.headers['api-key'] =  {YOUR_API_KEY}
delete opts.headers['Authorization']
delete opts.headers['OpenAI-Organization']

Ok but now the reason i'm here, offering help, looking for help. the way the stream responses are returned from Azure, is slightly different, i cant quite figure out what is happening, but it seems like it batches messages(events), so the data: [DONE] message is in with other messages. Its odd, because the deltas all come back as expected, but the fetch call never resolves, and as far as I can figure out its due to not seeing the done tag.

Working response events (each event wrapped in | |

Working snip from OpenAI API:

|data: {"id":"chatcmpl-7CoQaZkIbiDwtK6aU20z1UYtjUMrr","object":"chat.completion.chunk","created":1683288136,"model":"gpt-4-0314","choices":[{"delta":{"content":" today"},"index":0,"finish_reason":null}]}

|
|data: {"id":"chatcmpl-7CoQaZkIbiDwtK6aU20z1UYtjUMrr","object":"chat.completion.chunk","created":1683288136,"model":"gpt-4-0314","choices":[{"delta":{"content":"?"},"index":0,"finish_reason":null}]}

|
|data: {"id":"chatcmpl-7CoQaZkIbiDwtK6aU20z1UYtjUMrr","object":"chat.completion.chunk","created":1683288136,"model":"gpt-4-0314","choices":[{"delta":{},"index":0,"finish_reason":"stop"}]}

data: [DONE]

|

Broken snip from Azure API, truncated beginning of single event cause its very long, note that there are many data respones:

data: {"id":"chatcmpl-7CoZfRfl1fzZbwtjElNxXEJfH1cwV","object":"chat.completion.chunk","created":1683288699,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":" you"}}],"usage":null}

data: {"id":"chatcmpl-7CoZfRfl1fzZbwtjElNxXEJfH1cwV","object":"chat.completion.chunk","created":1683288699,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":" today"}}],"usage":null}

data: {"id":"chatcmpl-7CoZfRfl1fzZbwtjElNxXEJfH1cwV","object":"chat.completion.chunk","created":1683288699,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":"?"}}],"usage":null}

data: {"id":"chatcmpl-7CoZfRfl1fzZbwtjElNxXEJfH1cwV","object":"chat.completion.chunk","created":1683288699,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":"stop","delta":{}}],"usage":null}

data: [DONE]
|

Any help would be great.

Also, note that DEBUG does not return raw messages, to gett this detail I had to add the following to the custom fetch override (which broadly breaks the library from working, but gets the logging needed).

res.body.on('readable', () => {
    let chunk
    while (null !== (chunk = res.body.read())) {
      console.log(`|${chunk.toString()}|`)
    }
  })

[PvanHengel]

Ok so plot thickens, I added the parser logic in to my test, and interesting the parser outputs a final done message when running against OpenAI but not against Azure...

{ type: 'event', id: undefined, event: undefined, data: '[DONE]' }

unclear to me why the eventsource-parser library isnt emitting the final done message....

To add, it looks like maybe its related to the fact that Azure ends with a single line break after the DONE message, and OpenAI ends with a double line break. I'm not super farmiliar with how event streams are parsed, but to me it seems that maybe the missing \n is breaking the spec? Any ideas?

[PvanHengel]

Ok so progress, got it running, but have to hack it quite a bit by listening for the 'end' event on the SSE stream:

body.on("end", () => {
      console.log("END!!!")
      feed("[DONE]\n\n");
    });

then fire a dummy [DONE] event to get the library to resolve, far from ideal and will require a patch / change to the library either way. So looking for a steer as far as the approach to pull these pieces together.

Will need to reach out to microsoft to understand why they don't send a double new line before ending the stream with the done message..

[joostgrunwald]

@transitive-bullshit @xuhaoruins You can fix this by not using completions but chat/completions instead:

POC: const url = https://{resource-name}.openai.azure.com/openai/deployments/{deployment-name}/chat/completions?api-version={date_preview_here}; const headers = { "Content-Type": "application/json", "api-key": ${this._apiKey} //Authorization: Bearer ${this._apiKey} };

You probably want to build a option for this and change this small piece of code based on that.

This fully worked three weeks ago :) still using it in our teams bot

[alfogo]

Ok so plot thickens, I added the parser logic in to my test, and interesting the parser outputs a final done message when running against OpenAI but not against Azure...

{ type: 'event', id: undefined, event: undefined, data: '[DONE]' }

unclear to me why the eventsource-parser library isnt emitting the final done message....

To add, it looks like maybe its related to the fact that Azure ends with a single line break after the DONE message, and OpenAI ends with a double line break. I'm not super farmiliar with how event streams are parsed, but to me it seems that maybe the missing \n is breaking the spec? Any ideas?

same issue here when using Azure OpenAI and eventsource-parser. Missing that final DONE message..

[transitive-bullshit]

This project is undergoing a major revamp; closing out old issues as part of the prep process.

The chatgpt package is pretty outdated at this point. I recommend that you use the openai package or the openai-fetch package instead.