search

dylanshine / openai-kit

A community Swift package used to interact with the OpenAI API
https://platform.openai.com/docs/api-reference
MIT License
710 stars 109 forks source link

In Chat API, setting stream to true gives a different json structure #15

Closed arthurgarzajr closed 1 year ago

arthurgarzajr commented 1 year ago

When I implemented Chat, I didn't test what settings stream to true would do. Well, stream is very nice! But it returns data in a different way and this SDK currently doesn't support that.

dylanshine commented 1 year ago

@arthurgarzajr Can you provide an example of the response?

jaysack commented 1 year ago

I'm facing the same issue. I'm not sure what's the response type when setting stream to true, but here's an example from both responses with Postman.

stream set to true stream set to false
jaysack commented 1 year ago

It looks like the Content-Type property changes to text/event-stream when stream is set to true. Do you guys know how to receive this data type in Swift by any chance?

stream set to true stream set to false
dylanshine commented 1 year ago

@jaysack I think we would need to add Event Source support to this framework in order to handle this use case. Would need to do more research before taking a stab at an implementation. Considering removing the stream parameter in the interim.

ronaldmannak commented 1 year ago

I can confirm streaming doesn't work. I wonder if streaming would be a solution to a separate issue I'm having. If you use AWS as a proxy (to hide the API key), you'll receive 504: gateway time errors every time OpenAI slows down (which, unfortunately, is very often). There's no easy solution, since AWS has hardcoded a 30 second timeout in the gateway. But streaming will almost certainly start responding within 30 seconds, which might solve the issue. And even if not, streaming would be nice option to have.

Edit: there is a second reason this maybe should be a high priority hot fix. GPT4 is slow compared to GPT3-turbo. So slow I find myself not willing to wait for the answer and switching to Twitter or other apps.

Edit 2: I haven't tried it, but looks like this project implemented streaming successfully and is also based on NIO. It might be useful looking at their implementation: https://github.com/alfianlosari/ChatGPTSwift

Edit 3: I tried out the demo app. Streaming works well on the mentioned library. https://github.com/alfianlosari/ChatGPTSwiftUI

cgaaf commented 1 year ago

I've implemented streaming successfully using vanilla UrlSession async/await on a personal project. I've never contributed to open source before but I'd be happy to see if I can contribute.

dylanshine commented 1 year ago

@cgaaf More than happy for you to take a stab at it!

dylanshine commented 1 year ago

@ronaldmannak https://github.com/dylanshine/openai-kit/pull/36 adds stream support, curious if I could get your thoughts before merging

ronaldmannak commented 1 year ago

@dylanshine oh wow, with 19 changed files it looks like a bigger change than I thought. Let me use the branch and see how it works.

dylanshine commented 1 year ago

@ronaldmannak A lot of it was replacing HTTPClient.Body in favor of Data to be more flexible

dylanshine commented 1 year ago

PR merged...have fun