Use of Azure endpoints?

iainmwallace commented 1 year ago

This is really awesome. Is it possible to use with Azure endpoints?

https://learn.microsoft.com/en-us/azure/cognitive-services/openai/quickstart?pivots=programming-language-python&tabs=command-line

Thanks

wch commented 1 year ago

Yes, it should be possible, although I have not tested it with Azure myself.

When you call chatstream.chat_server(), you can pass it a url. See the docs here: https://wch.github.io/chatstream/reference/chat_server.html#chatstream.chat_server

I believe you should still set the OPENAI_API_KEY environment variable. In the docs you linked to, they recommend setting AZURE_OPENAI_KEY, which you could do, but then you would need to get that env var's value manually and pass it like chatstream.chat_server(api_key=os.getenv("AZURE_OPENAI_KEY"), ...).

If you test it out, please let me know how it goes!

iainmwallace commented 1 year ago

I tried but it didn't seem to work - perhaps due to company firewall. I will dig in further

trangdata commented 1 year ago

Hi @wch, any chance you can add proxy and deployment_name as arguments to chat_server? There have been some issues regarding discrepancies between model and engine/deployment names on the Azure endpoint.

I modified my server function in basic/app.py as follows:

def server(input: Inputs, output: Outputs, session: Session):
    chatstream.chat_server(
        "mychat",
        model="Turbo", # also tried "gpt-35-turbo", didn't work
        url=os.getenv("OPENAI_API_BASE"),
        api_key=os.getenv("OPENAI_API_KEY"),
    )

and getting

Unhandled error: 'Could not automatically map Turbo to a tokeniser. Please use tiktok.get_encoding to explicitly get the tokeniser you expect.'

When I set model="gpt-3.5-turbo", I got

Unhandled error: Please set the OPENAI_API_KEY environment variable to your OpenAI API key. which sounds like it's still using the default Open AI url, not my Azure endpoint.

I'm not sure if we're using langchain but I wanted to include a few issues in case they're relevant:

iainmwallace commented 1 year ago

In case it is useful information gptstudio provides the ability to connect to an Azure endpoint.

https://github.com/MichelNivard/gptstudio

wch commented 1 year ago

@trangdata I don't have access to Azure OpenAI, but I just applied for access. Hopefully I will be given access in the near future, but until then I'm not able to test anything with Azure.

I'll try adding proxy and deployment_name in a pull request.

For reference, this is their documentation on the differences between using OpenAI's endpoints vs. Azure's endpoints. https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/switching-endpoints

wch commented 1 year ago

@trangdata I don't see proxy as a possible parameter to .create() or .acreate(). Taking a quick look at the openai source code, I think you will have to set it like this:

import openai

openai.proxy = "..."

https://github.com/openai/openai-python/blob/b82a3f7e4c462a8a10fa445193301a3cefef9a4a/openai/__init__.py#L56

wch commented 1 year ago

I've created a PR for Azure support, #10. Please see that PR and test it out. There is more information and instructions in the PR.

trangdata commented 1 year ago

Thank you for the quick response! I installed the new chatstream version in the PR, added some openai configs (below) to my app.py, and uses endpoint_type = "azure" in chat_server, and it doesn't error out but the app just kinda stalls with ... in the response text box.

openai.api_type = os.getenv("OPENAI_API_TYPE")
openai.api_base = os.getenv("OPENAI_API_BASE")
openai.api_key = os.getenv("OPENAI_API_KEY")
openai.proxy = os.getenv("OPENAI_PROXY")
openai.api_version = "2023-03-15-preview"  # "2022-12-01"

When I set model="Turbo" or model="gpt-35-turbo", which is what I would do for other projects using the Azure endpoint, I received the same error as before.

Unhandled error: 'Could not automatically map gpt-35-turbo to a tokeniser. Please use tiktok.get_encoding to explicitly get the tokeniser you expect.'

wch commented 1 year ago

Quick answer because I’m on my phone: shouldn’t there be a period in gpt-3.5-turbo?

trangdata commented 1 year ago

Actually no. Azure is very odd with their engine/deployment name and model name. See https://github.com/hwchase17/langchain/issues/1577#issuecomment-1479489910 and example: https://github.com/hwchase17/langchain/issues/3251#issuecomment-1566531223

wch commented 1 year ago

In the PR I just pushed some code to map from Azure model names to OpenAI model names. @trangdata Please give it another shot. Thanks!

trangdata commented 1 year ago

Thanks, Winston!

So this time, instead of erroring out immediately, the app just kinda stalls forever. And when I Ctrl C, I see:

Task exception was never retrieved future: <Task finished name='Task-9' coro=<stream_to_reactive..task_main() done, defined at /Users/let20/Documents/chatstream/chatstream/init.py:547> exception=InvalidRequestError(message="Must provide an 'engine' or 'deployment_id' parameter to create a <class 'openai.api_resources.chat_completion.ChatCompletion'>", param='engine', code=None, http_status=None, request_id=None)>

Full message

(chatstream) ➜ chatstream shiny run examples/basic/app.py --launch-browser INFO: Started server process [19238] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit) INFO: 127.0.0.1:58174 - "GET / HTTP/1.1" 200 OK INFO: ('127.0.0.1', 58175) - "WebSocket /websocket/" [accepted] INFO: connection open ^CINFO: Shutting down INFO: connection closed INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [19238] Task exception was never retrieved future: .task_main() done, defined at /Users/let20/Documents/chatstream/chatstream/__init__.py:547> exception=InvalidRequestError(message="Must provide an 'engine' or 'deployment_id' parameter to create a ", param='engine', code=None, http_status=None, request_id=None)> Traceback (most recent call last): File "/Users/let20/Documents/chatstream/chatstream/__init__.py", line 550, in task_main func = await func # type: ignore ^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_resources/chat_completion.py", line 45, in acreate return await super().acreate(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 214, in acreate ) = cls.__prepare_create_request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 83, in __prepare_create_request raise error.InvalidRequestError( openai.error.InvalidRequestError: Must provide an 'engine' or 'deployment_id' parameter to create a

wch commented 1 year ago

I received access to Azure OpenAI, and got it working; see #10 again for instructions and an example app. Please give it a shot!

The API in that PR is currently really awkward to accommodate OpenAI and Azure-OpenAI, and I will clean that up before merging.

trangdata commented 1 year ago

Hmm, so I installed the new changes, but the app hangs after I enter a prompt.

openai.error.APIConnectionError: Error communicating with OpenAI

Full message

Task exception was never retrieved future: .task_main() done, defined at /Users/let20/Documents/chatstream/chatstream/__init__.py:547> exception=APIConnectionError(message='Error communicating with OpenAI', http_status=None, request_id=None)> Traceback (most recent call last): File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py", line 668, in arequest_raw result = await session.request(**request_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/client.py", line 536, in _request conn = await self._connector.connect( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/connector.py", line 540, in connect proto = await self._create_connection(req, traces, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/connector.py", line 899, in _create_connection _, proto = await self._create_proxy_connection(req, traces, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/connector.py", line 1221, in _create_proxy_connection proxy_req = ClientRequest( ^^^^^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 305, in __init__ self.update_host(url) File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 364, in update_host raise InvalidURL(url) aiohttp.client_exceptions.InvalidURL: proxy-server.bms.com:8080 The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/let20/Documents/chatstream/chatstream/__init__.py", line 550, in task_main func = await func # type: ignore ^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_resources/chat_completion.py", line 45, in acreate return await super().acreate(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 217, in acreate response, _, api_key = await requestor.arequest( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py", line 372, in arequest result = await self.arequest_raw( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py", line 685, in arequest_raw raise error.APIConnectionError("Error communicating with OpenAI") from e openai.error.APIConnectionError: Error communicating with OpenAI

iainmwallace commented 1 year ago

Is there any way of testing the api call is able to connect?

I am also getting the hanging issue but I get an SSL error saying that SSL:certificate_verify_failed for our Azure endpoint.

Thanks!

wch commented 1 year ago

@iainmwallace I have a had some difficultly getting error messages to show up immediately. I think it has something to do with it receiving the data on a separate Task, but I'm not 100% sure.

@trangdata I see some references to a proxy in the stack trace, and I know you mentioned something about a proxy earlier. I don't see anything in the Azure OpenAI docs about using a proxy, though. How are you using it?

For both of you, have you tried running the example app in https://github.com/wch/chatstream/pull/10#issuecomment-1641451668 ?

Note that you will need to set the openai.api_base URL (https://winstontest.openai.azure.com in the example) as well as a deployment ID (my-gpt-35-turbo in the example).

wch commented 1 year ago

Here is a simple script I used to test communication with Azure:

import openai
openai.api_type = "azure"
openai.api_base = "https://winstontest.openai.azure.com/"  # <- Customize this
openai.api_version = "2023-03-15-preview"
openai.api_key = "xxxxxxx"                                 # <- Customize this

resp = await openai.ChatCompletion.acreate(
  deployment_id="my-gpt-35-turbo",                         # <- Customize this
  messages = [{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"Tell me about yourself"}],
  stream=True
)

print(resp)

async for chunk in resp:
    print(chunk)

Note that this must be run with an interpreter that is run using asyncio, like ipython or (I think) Jupyter. If you use the plain python interpreter, it will give this error: SyntaxError: 'await' outside function.

If you run this, it should print out a lot of messages which look something like this:

{
  "id": "chatcmpl-7e7JqYddw8TgaKtbzs6VTwqk1rE5R",
  "object": "chat.completion.chunk",
  "created": 1689795610,
  "model": "gpt-35-turbo",
  "choices": [
    {
      "index": 0,
      "finish_reason": null,
      "delta": {
        "content": " efficient"
      }
    }
  ],
  "usage": null
}
{
  "id": "chatcmpl-7e7JqYddw8TgaKtbzs6VTwqk1rE5R",
  "object": "chat.completion.chunk",
  "created": 1689795610,
  "model": "gpt-35-turbo",
  "choices": [
    {
      "index": 0,
      "finish_reason": null,
      "delta": {
        "content": "."
      }
    }
  ],
  "usage": null
}
{
  "id": "chatcmpl-7e7JqYddw8TgaKtbzs6VTwqk1rE5R",
  "object": "chat.completion.chunk",
  "created": 1689795610,
  "model": "gpt-35-turbo",
  "choices": [
    {
      "index": 0,
      "finish_reason": "stop",
      "delta": {}
    }
  ],
  "usage": null
}

If this works, I think the example app in the PR should work.

iainmwallace commented 1 year ago

Thanks, that is incredibly helpful.

I get an errror 'InvalidURL' when I use the function acreate:

   resp = await openai.ChatCompletion.acreate(model="gpt-4", deployment_id="gpt-4", messages=[{"role": "user", "content": "Hello world"}])

But with the chatcompletion function example from https://github.com/openai/openai-python#usage, it works as expected:

  chat_completion = openai.ChatCompletion.create(model="gpt-4", deployment_id="gpt-4",messages=[{"role": "user", "content": "Hello world"}])

wch commented 1 year ago

A couple questions:

What version of openai are you using?

>>> import openai
>>> openai.__version__
'0.27.8'

What happens if you call acreate() without model? My understanding is that the model parameter isn't used by Azure, and perhaps including it doesn't cause problems for create(), but does cause problems for acreate(). See here for the docs about it: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/switching-endpoints#keyword-argument-for-model

iainmwallace commented 1 year ago

I am using 0.27.8 and I get the same error when removing the model parameter in acreate(). Removing create() continues to work without the model parameter.

I will ask internally if anyone has gotten this function to work, and will report back

wch commented 1 year ago

I don't think you're running into this problem, but for future reference, I just realized that I need to mention one very important thing about the acreate() code I provided -- it must be run in an environment that's run with asyncio so that await can be called from the top level. I'll edit the comment above to mention this.

In practice, I ran it with ipython. (I think it should work from a Jupyter notebook as well but I'm not 100% sure.)

If you run it with the plain python interpreter, it will give this error when you use await at the top level. For example:

>>> resp = await openai.ChatCompletion.acreate( .... )
SyntaxError: 'await' outside function

nsvbhat commented 12 months ago

I am trying to use it in conjunction with azure openai... did try passing the params to chatstream.chat_server( "mychat", url=config['AZURE_OPENAI_BASE_URL'], api_key=config['AZURE_OPENAI_API_KEY'], model="gpt-4", text_input_placeholder="How can I help you?", deployment_id="gpt-4", debug=True )
but I get an error stating

and if I remove the deployment_id, I am getting the error stating engine or deployment_id missing

am I missing something??

trangdata commented 12 months ago

@nsvbhat Did you install the chatstream version on that PR?

pip install chatstream@git+https://github.com/wch/chatstream.git@azure

nsvbhat commented 12 months ago

@trangdata yes.. i have used the same one..

trangdata commented 12 months ago

@nsvbhat oh I think azure_deployment_id is the argument you want, not deployment_id.

nsvbhat commented 12 months ago

ok i see that in the library its called azure_deployment_id and not deployment_id .. will trying using that n see.

trangdata commented 12 months ago

@trangdata I see some references to a proxy in the stack trace, and I know you mentioned something about a proxy earlier. I don't see anything in the Azure OpenAI docs about using a proxy, though. How are you using it?

I'm setting proxy in app.py along with other openai variables, and this has worked for me in different projects too:

openai.api_type = os.getenv("OPENAI_API_TYPE")
openai.api_base = os.getenv("OPENAI_API_BASE")
openai.api_key = os.getenv("OPENAI_API_KEY")
openai.proxy = "proxy-server.bms.com:8080"
openai.api_version = "2023-03-15-preview"

For both of you, have you tried running the example app in https://github.com/wch/chatstream/pull/10#issuecomment-1641451668 ? Note that you will need to set the openai.api_base URL (https://winstontest.openai.azure.com in the example) as well as a deployment ID (my-gpt-35-turbo in the example).

Yes. This is what my server function looks like:

def server(input: Inputs, output: Outputs, session: Session):
    chatstream.chat_server(
        "mychat",
        api_key=os.getenv("OPENAI_API_KEY"),
        model="gpt-3.5-turbo",
        azure_deployment_id="Turbo",
        debug=True,
    )

>>> import openai
>>> openai.__version__
'0.27.8'

I have the exact same experience with @iainmwallace when I tested for communication with Azure using acreate (running on vscode interpreter using jupyter). But things are fine with create:

response = openai.ChatCompletion.create(
    engine="Turbo", 
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Tell me about yourself"},
    ],
    temperature=0.7
)
response

<OpenAIObject chat.completion id=chatcmpl-7elyKFl8th1wx0tvBT2B1MbgIVnwm at 0x109497cb0> JSON: {
  "id": "chatcmpl-7elyKFl8th1wx0tvBT2B1MbgIVnwm",
  "object": "chat.completion",
  "created": 1689951880,
  "model": "gpt-35-turbo",
  "choices": [
    {
      "index": 0,
      "finish_reason": "stop",
      "message": {
        "role": "assistant",
        "content": "As an AI assistant, I am designed to provide helpful information and assist with various tasks. I can answer questions, provide recommendations, assist with scheduling, and more. I am constantly learning and updating my knowledge base to better assist users like you. How can I help you today?"
      }
    }
  ],
  "usage": {
    "completion_tokens": 56,
    "prompt_tokens": 21,
    "total_tokens": 77
  }
}

Error with acreate()

InvalidURL Traceback (most recent call last) File [~/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py:668](https://file+.vscode-resource.vscode-cdn.net/Users/let20/Documents/chatstream/~/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py:668), in APIRequestor.arequest_raw(self, method, url, session, params, supplied_headers, files, request_id, request_timeout) [667](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py?line=666) try: --> [668](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py?line=667) result = await session.request(**request_kwargs) [669](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py?line=668) util.log_info( [670](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py?line=669) "OpenAI API response", [671](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py?line=670) path=abs_url, (...) [674](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py?line=673) request_id=result.headers.get("X-Request-Id"), [675](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py?line=674) ) File [~/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/client.py:536](https://file+.vscode-resource.vscode-cdn.net/Users/let20/Documents/chatstream/~/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/client.py:536), in ClientSession._request(self, method, str_or_url, params, data, json, cookies, headers, skip_auto_headers, auth, allow_redirects, max_redirects, compress, chunked, expect100, raise_for_status, read_until_eof, proxy, proxy_auth, timeout, verify_ssl, fingerprint, ssl_context, ssl, proxy_headers, trace_request_ctx, read_bufsize) [535](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/client.py?line=534) assert self._connector is not None --> [536](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/client.py?line=535) conn = await self._connector.connect( [537](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/client.py?line=536) req, traces=traces, timeout=real_timeout [538](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/client.py?line=537) ) [539](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/client.py?line=538) except asyncio.TimeoutError as exc: File [~/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/connector.py:540](https://file+.vscode-resource.vscode-cdn.net/Users/let20/Documents/chatstream/~/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/connector.py:540), in BaseConnector.connect(self, req, traces, timeout) [539](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/connector.py?line=538) try: --> [540](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/connector.py?line=539) proto = await self._create_connection(req, traces, timeout) [541](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/connector.py?line=540) if self._closed: File [~/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/connector.py:899](https://file+.vscode-resource.vscode-cdn.net/Users/let20/Documents/chatstream/~/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/connector.py:899), in TCPConnector._create_connection(self, req, traces, timeout) ... [683](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py?line=682) raise error.Timeout("Request timed out") from e [684](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py?line=683) except aiohttp.ClientError as e: --> [685](file:///Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py?line=684) raise error.APIConnectionError("Error communicating with OpenAI") from e APIConnectionError: Error communicating with OpenAI Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?5be4fd83-9e1b-449c-bfa9-572da7e15741) or open in a [text editor](command:workbench.action.openLargeOutput?5be4fd83-9e1b-449c-bfa9-572da7e15741). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...

nsvbhat commented 11 months ago

@trangdata I tried using acreate and seems to be working with minor additions as per this link while using acreate..

`import openai from aiohttp import ClientSession

openai.api_type = "azure" openai.api_base = "your endpoint" openai.api_version = "2023-05-15" openai.api_key = "user key" openai.proxy= "//proxy-server.bms.com:8080"

messages = [ {"role":"system","content":"You are a helpful assistant."}, {"role":"user","content":"Tell me about yourself"} ]

async def main(): openai.aiosession.set(ClientSession()) resp = await openai.ChatCompletion.acreate( model="gpt-4", deployment_id="gpt-4", messages=messages, ) print(resp) await openai.aiosession.get().close()

import asyncio asyncio.run(main())`

Output:

but the same thing does not work for chatstream :(

trangdata commented 11 months ago

Thank you so much @wch and @nsvbhat for your help. I finally was able to deploy the app. Two things:

I needed to use the full proxy URL as "http://proxy-server.bms.com:8080" or at least add "//" in front of my usual proxy: "//proxy-server.bms.com:8080"
The model argument must be the actual OpenAI model. So even though Azure uses "gpt-35-turbo", we still have to specify model="gpt-3.5-turbo" (with the dot) in our function. This is not an issue if we use another model like "gpt-4".

Here's the entirety of my app:

import os

import openai
from dotenv import load_dotenv
from shiny import App, Inputs, Outputs, Session, ui

import chatstream

load_dotenv()

openai.api_type = os.getenv("OPENAI_API_TYPE")
openai.api_base = os.getenv("OPENAI_API_BASE")
openai.api_key = os.getenv("OPENAI_API_KEY")
openai.proxy = "http://proxy-server.bms.com:8080"
openai.api_version = "2023-03-15-preview"

app_ui = ui.page_fixed(
    chatstream.chat_ui("mychat"),
)

def server(input: Inputs, output: Outputs, session: Session):
    chatstream.chat_server(
        "mychat",
        api_key=os.getenv("OPENAI_API_KEY"),
        model="gpt-3.5-turbo",
        azure_deployment_id="Turbo",
        # model="gpt-4", # also works with GPT-4 as deployment name
        # azure_deployment_id="GPT-4",
        debug=True,
    )

app = App(app_ui, server)

I think we can merge #10 now @wch.

trangdata commented 11 months ago

A final note: @wch's snippet for testing connection was really helpful. Again, this works when my proxy has "//" in front but errors out otherwise.

import openai

openai.api_type = os.getenv("OPENAI_API_TYPE")
openai.api_base = os.getenv("OPENAI_API_BASE")
openai.api_key = os.getenv("OPENAI_API_KEY")
openai.api_version = "2023-03-15-preview"
openai.proxy = "//proxy-server.bms.com:8080"

resp = await openai.ChatCompletion.acreate(
    deployment_id="Turbo",
    model="gpt-35-turbo",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Tell me about yourself"},
    ],
    stream=True,
)

print(resp)
async for chunk in resp:
    print(chunk)

wch commented 11 months ago

Great, I'm glad to know that it is working! Before merging #10, I'm planning on making some improvements to the API.

filipwastberg commented 9 months ago

Hi!

I can connect to the Azure API and get a response.

import openai
import os
openai.api_type = "azure"
openai.api_base = "https://chatgptustest.openai.azure.com/"
openai.api_version = "2023-07-01-preview"
openai.api_key = os.getenv("OPENAI_API_KEY")

response = openai.ChatCompletion.create(
    engine="gpt-4-32k", 
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Tell me about yourself"},
    ],
    temperature=0.7
)
print(response)

Response

``` {... "created": 1697460801, "model": "gpt-4-32k", "prompt_filter_results": [ { "prompt_index": 0, ... "choices": [ { "index": 0, "finish_reason": "stop", "message": { "role": "assistant", "content": "I am an AI language model, designed to assist users with their questions, provide information, and engage in conversation. I am constantly learning and updating my knowledge base to better assist you. I can help with a wide range of topics, from general knowledge and trivia to problem-solving and advice." ```

But when trying to run the example above:

import os
import openai
from shiny import App, Inputs, Outputs, Session, ui
import chatstream

openai.api_type = "azure"
openai.api_base = "https://chatgptustest.openai.azure.com/"
openai.api_version = "2023-07-01-preview"
openai.api_key = os.getenv("OPENAI_API_KEY")

app_ui = ui.page_fixed(
    chatstream.chat_ui("mychat"),
)

def server(input: Inputs, output: Outputs, session: Session):
    chatstream.chat_server(
        "mychat",
        api_key=os.getenv("OPENAI_API_KEY"),
        model="gpt-4-32k",
        azure_deployment_id="gpt-4-32k",
        debug=True,
    )

app = App(app_ui, server)

The app starts and I can write a message. However, it fails when I apply the message and I get the following error message:

Error message

``` Traceback (most recent call last): File "/Users/filip.wastberg/Work/shiny-python/azure-open-ai-shiny/shinyenv/lib/python3.10/site-packages/shiny/reactive/_reactives.py", line 548, in _run await self._fn() File "/Users/filip.wastberg/Work/shiny-python/azure-open-ai-shiny/shinyenv/lib/python3.10/site-packages/shiny/reactive/_reactives.py", line 823, in new_user_async_fn return await user_fn() File "/Users/filip.wastberg/Work/shiny-python/azure-open-ai-shiny/shinyenv/lib/python3.10/site-packages/chatstream/__init__.py", line 265, in finalize_streaming_result if "content" in message["choices"][0]["delta"]: IndexError: list index out of range /Users/filip.wastberg/Work/shiny-python/azure-open-ai-shiny/shinyenv/lib/python3.10/site-packages/shiny/reactive/_reactives.py:530: ReactiveWarning: Error in Effect: list index out of range await self._run() ```

Any idea what might cause this? Looks lika a Shiny problem?

I have installed chatstream via pip install chatstream@git+https://github.com/wch/chatstream.git@azure

This is what my environment looks like:

requirements.txt

``` aiohttp==3.8.6 aiosignal==1.3.1 anyio==4.0.0 appdirs==1.4.4 appnope==0.1.3 asgiref==3.7.2 asttokens==2.4.0 async-timeout==4.0.3 attrs==23.1.0 backcall==0.2.0 certifi==2023.7.22 charset-normalizer==3.3.0 chatstream==0.0.1 click==8.1.7 comm==0.1.4 debugpy==1.8.0 decorator==5.1.1 docopt==0.6.2 exceptiongroup==1.1.3 executing==2.0.0 frozenlist==1.4.0 h11==0.14.0 htmltools==0.3.0.9000 idna==3.4 ipykernel==6.25.2 ipython==8.16.1 jedi==0.19.1 jupyter_client==8.4.0 jupyter_core==5.4.0 linkify-it-py==2.0.2 markdown-it-py==3.0.0 matplotlib-inline==0.1.6 mdit-py-plugins==0.4.0 mdurl==0.1.2 multidict==6.0.4 nest-asyncio==1.5.8 openai==0.28.1 packaging==23.2 parso==0.8.3 pexpect==4.8.0 pickleshare==0.7.5 pipreqs==0.4.13 platformdirs==3.11.0 prompt-toolkit==3.0.39 psutil==5.9.6 ptyprocess==0.7.0 pure-eval==0.2.2 Pygments==2.16.1 python-dateutil==2.8.2 python-multipart==0.0.6 pyzmq==25.1.1 regex==2023.10.3 requests==2.31.0 shiny==0.5.1.9003 six==1.16.0 sniffio==1.3.0 stack-data==0.6.3 starlette==0.31.1 tiktoken==0.5.1 tornado==6.3.3 tqdm==4.66.1 traitlets==5.11.2 typing_extensions==4.8.0 uc-micro-py==1.0.2 urllib3==2.0.6 uvicorn==0.23.2 watchfiles==0.21.0 wcwidth==0.2.8 websockets==11.0.3 yarg==0.1.9 yarl==1.9.2 ```

wch / chatstream

Use of Azure endpoints? #8