Closed ramzeng closed 1 year ago
My environment is (Nginx + PHP-FPM + Laravel)
$stream = $client->chat()->createStreamed($body);
it always waits for all data return from 'openai'.
but if i use the php artisan serve
, it will return soon.
Could you give me some advice? thank you very much.
Sorry, no idea.
My environment is (Nginx + PHP-FPM + Laravel)
$stream = $client->chat()->createStreamed($body);
it always waits for all data return from 'openai'.
but if i use the
php artisan serve
, it will return soon.
Having the same issue
Are you trying to stream it directly to a web browser or calling it from another interface?
On Sun, May 7, 2023 at 2:59 PM Charlie Joseph @.***> wrote:
My environment is (Nginx + PHP-FPM + Laravel)
$stream = $client->chat()->createStreamed($body);
it always waits for all data return from 'openai'.
but if i use the php artisan serve, it will return soon.
Having the same issue
— Reply to this email directly, view it on GitHub https://github.com/openai-php/client/issues/100#issuecomment-1537549373, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEI6P74XTT2LLAZYY4EELLXFALKJANCNFSM6AAAAAAWXLTOFU . You are receiving this because you are subscribed to this thread.Message ID: @.***>
Kind Regards, Ryan Decker
@. @.> | 760.459.RYAN (7926)*
I want an api endpoint which returns the stream, then my vue site fetches the stream, and returns it on the site as its coming in (similar to real site where the words come in one by one)
May want to drop the chunks in a database or somewhere else. I haven't had great luck streaming from PHP. Have you considered server side messages?
On Sun, May 7, 2023 at 4:37 PM Charlie Joseph @.***> wrote:
I want an api endpoint which returns the stream, then my vue site fetches the stream, and returns it on the site as its coming in (similar to real site where the words come in one by one)
— Reply to this email directly, view it on GitHub https://github.com/openai-php/client/issues/100#issuecomment-1537568504, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEI6P55EKSUHBXMMZWYN4LXFAW2RANCNFSM6AAAAAAWXLTOFU . You are receiving this because you commented.Message ID: @.***>
Kind Regards, Ryan Decker
@. @.> | 760.459.RYAN (7926)*
May want to drop the chunks in a database or somewhere else. I haven't had great luck streaming from PHP. Have you considered server side messages? …
I was having the same frustration tbh.. by messages meaning like web sockets?
I did consider but haven’t tried, in case I was doing something wrong haha
Could you give me some advice? thank you very much.
FPM的问题,我准备换swoole了。兄弟你解决了么?😂
Try using these headers in stream()
call:
return response()->stream(function () {
....
}, 200, [
'Cache-Control' => 'no-cache, must-revalidate',
'Content-Type' => 'text/event-stream',
'X-Accel-Buffering' => 'no',
]);
Try using these headers in
stream()
call:return response()->stream(function () { .... }, 200, [ 'Cache-Control' => 'no-cache, must-revalidate', 'Content-Type' => 'text/event-stream', 'X-Accel-Buffering' => 'no', ]);
do not work with FPM mode
must-revalidate
is 'must-revalidate' required?
do not work with FPM mode
And nginx? It worked for me with Caddy & PHP-FPM and Caddy & Swoole. I think it’s likely nginx buffering the response, assuming it works through artisan serve.
do not work with FPM mode
And nginx? It worked for me with Caddy & PHP-FPM and Caddy & Swoole. I think it’s likely nginx buffering the response, assuming it works through artisan serve.
works but not response one by one which response many chunks together,under cli mode ,everything is fine,but only one sse connection under cli mode aka artisan serve
Yeah, so it seems like nginx is buffering output. Supposedly X-Accel-Buffering
is supposed to prevent that, but don't know enough about nginx to be more help. As an experiment could look into disabling or reducing size of nginx buffers through config, but not sure that's a good long-term solution.
Your webserver has to support the behavior. I ran into the same issue. flush/ob_flush etc (Apache) wasn't enough. I had to add config for apache:
<Proxy "fcgi://php:9000/">
ProxySet enablereuse=on flushpackets=on max=10
</Proxy>
YMMV on the above settings, I am doing local development. After that, doing this from a controller worked:
$data = json_decode($request->getContent());
$stream = $this->client->completions()->createStreamed(
[
'model' => $data->options->model ?? 'text-davinci-003',
'prompt' => trim($data->prompt),
'temperature' => floatval($data->options->temperature),
'max_tokens' => (int) $data->options->max_tokens,
]
);
return new StreamedResponse(function () use ($stream) {
foreach ($stream as $data) {
echo $data->choices[0]->text;
ob_flush();
flush();
}
}, 200, ['Content-Type' => 'text/plain']);
You can see the result here where I am streaming a response to a CKEditor 5 plugin:
https://www.youtube.com/watch?v=8pR-gi4jAek
If your server isn't configured properly, it will just wait until the entire response is ready, no matter if you are using createStreamed
or not. I can't speak to nginx though as I do not use that.
Throw the streamed chunk content into DB and fetch it via ajax on the frontend.
This is my approach if anyone is curious: https://twitter.com/staticmaker1/status/1662729320341852160
any solutions for nginx found?
EDIT: nginx sites enabled->*conf
location ~ \.php$ {
# Other directives...
# Disable buffering for response streams
fastcgi_buffering off;
# Optional: Adjust buffer sizes for streaming
fastcgi_buffers 16 16k;
fastcgi_buffer_size 32k;
# Other directives...
}
seems to solved this issue to me
i sloved it too, focus on php-fpm cache settings
fastcgi_buffers
it's work