something wrong with stream mode

[ramzeng]

/** @var Client $client */
$client = app(Client::class);

$body = [
    ...
    ...
    'messages' => [
        [
            'role' => 'user',
            'content' => 'Hello',
        ]
    ],
];

$stream = $client->chat()->createStreamed($body);

return response()->stream(function () use ($stream) {
    foreach ($stream as $response) {
        $choice = Arr::first($response->choices);

        if (empty($choice->delta->content)) {
            continue;
        }

        echo $choice->delta->content;
        ob_flush();
        flush();
    }
}, 200, ['X-Accel-Buffering' => 'no'])

[ramzeng]

My environment is (Nginx + PHP-FPM + Laravel)

$stream = $client->chat()->createStreamed($body);

it always waits for all data return from 'openai'.

but if i use the php artisan serve, it will return soon.

[ramzeng]

Could you give me some advice? thank you very much.

[nunomaduro]

Sorry, no idea.

[heychazza]

My environment is (Nginx + PHP-FPM + Laravel)

$stream = $client->chat()->createStreamed($body);

it always waits for all data return from 'openai'.

but if i use the php artisan serve, it will return soon.

Having the same issue

[edjamakated]

Are you trying to stream it directly to a web browser or calling it from another interface?

On Sun, May 7, 2023 at 2:59 PM Charlie Joseph @.***> wrote:

My environment is (Nginx + PHP-FPM + Laravel)

$stream = $client->chat()->createStreamed($body);

it always waits for all data return from 'openai'.

but if i use the php artisan serve, it will return soon.

Having the same issue

— Reply to this email directly, view it on GitHub https://github.com/openai-php/client/issues/100#issuecomment-1537549373, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEI6P74XTT2LLAZYY4EELLXFALKJANCNFSM6AAAAAAWXLTOFU . You are receiving this because you are subscribed to this thread.Message ID: @.***>

--

Kind Regards, Ryan Decker

@. @.> | 760.459.RYAN (7926)*

[heychazza]

I want an api endpoint which returns the stream, then my vue site fetches the stream, and returns it on the site as its coming in (similar to real site where the words come in one by one)

[edjamakated]

May want to drop the chunks in a database or somewhere else. I haven't had great luck streaming from PHP. Have you considered server side messages?

On Sun, May 7, 2023 at 4:37 PM Charlie Joseph @.***> wrote:

I want an api endpoint which returns the stream, then my vue site fetches the stream, and returns it on the site as its coming in (similar to real site where the words come in one by one)

— Reply to this email directly, view it on GitHub https://github.com/openai-php/client/issues/100#issuecomment-1537568504, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEI6P55EKSUHBXMMZWYN4LXFAW2RANCNFSM6AAAAAAWXLTOFU . You are receiving this because you commented.Message ID: @.***>

--

Kind Regards, Ryan Decker

@. @.> | 760.459.RYAN (7926)*

[heychazza]

May want to drop the chunks in a database or somewhere else. I haven't had great luck streaming from PHP. Have you considered server side messages? …

I was having the same frustration tbh.. by messages meaning like web sockets?

I did consider but haven’t tried, in case I was doing something wrong haha

[pensacola1989]

Could you give me some advice? thank you very much.

FPM的问题，我准备换swoole了。兄弟你解决了么？😂

[pb30]

Try using these headers in stream() call:

return response()->stream(function () {
    ....
}, 200, [
    'Cache-Control'     => 'no-cache, must-revalidate',
    'Content-Type'      => 'text/event-stream',
    'X-Accel-Buffering' => 'no',
]);

[pensacola1989]

Try using these headers in stream() call:

return response()->stream(function () {
    ....
}, 200, [
    'Cache-Control'     => 'no-cache, must-revalidate',
    'Content-Type'      => 'text/event-stream',
    'X-Accel-Buffering' => 'no',
]);

do not work with FPM mode

[pensacola1989]

must-revalidate

is 'must-revalidate' required?

[pb30]

do not work with FPM mode

And nginx? It worked for me with Caddy & PHP-FPM and Caddy & Swoole. I think it’s likely nginx buffering the response, assuming it works through artisan serve.

[pensacola1989]

do not work with FPM mode

And nginx? It worked for me with Caddy & PHP-FPM and Caddy & Swoole. I think it’s likely nginx buffering the response, assuming it works through artisan serve.

works but not response one by one which response many chunks together，under cli mode ，everything is fine，but only one sse connection under cli mode aka artisan serve

[pb30]

Yeah, so it seems like nginx is buffering output. Supposedly X-Accel-Buffering is supposed to prevent that, but don't know enough about nginx to be more help. As an experiment could look into disabling or reducing size of nginx buffers through config, but not sure that's a good long-term solution.

[kevinquillen]

Your webserver has to support the behavior. I ran into the same issue. flush/ob_flush etc (Apache) wasn't enough. I had to add config for apache:

<Proxy "fcgi://php:9000/">
  ProxySet enablereuse=on flushpackets=on max=10
</Proxy>

YMMV on the above settings, I am doing local development. After that, doing this from a controller worked:

    $data = json_decode($request->getContent());

    $stream = $this->client->completions()->createStreamed(
      [
        'model' => $data->options->model ?? 'text-davinci-003',
        'prompt' => trim($data->prompt),
        'temperature' => floatval($data->options->temperature),
        'max_tokens' => (int) $data->options->max_tokens,
      ]
    );

    return new StreamedResponse(function () use ($stream) {
      foreach ($stream as $data) {
        echo $data->choices[0]->text;
        ob_flush();
        flush();
      }
    }, 200, ['Content-Type' => 'text/plain']);

You can see the result here where I am streaming a response to a CKEditor 5 plugin:

https://www.youtube.com/watch?v=8pR-gi4jAek

If your server isn't configured properly, it will just wait until the entire response is ready, no matter if you are using createStreamed or not. I can't speak to nginx though as I do not use that.

[dilab]

Throw the streamed chunk content into DB and fetch it via ajax on the frontend.

[dilab]

This is my approach if anyone is curious: https://twitter.com/staticmaker1/status/1662729320341852160

[pavelthq]

any solutions for nginx found?

EDIT: nginx sites enabled->*conf

location ~ \.php$ {
    # Other directives...

    # Disable buffering for response streams
    fastcgi_buffering off;

    # Optional: Adjust buffer sizes for streaming
    fastcgi_buffers 16 16k;
    fastcgi_buffer_size 32k;

    # Other directives...
}

seems to solved this issue to me

[ramzeng]

i sloved it too, focus on php-fpm cache settings

[QThans]

fastcgi_buffers

it's work