Open maxduke opened 1 month ago
Dean-YZG
commented
1 month ago The timeout is currently set to 60 seconds in nextchat, you might want to check if something else is causing the 25-second prompt to run out, Or do you have any error screenshots? Let's examine the causes of early timeouts
maxduke
commented
1 month ago The timeout is currently set to 60 seconds in nextchat, you might want to check if something else is causing the 25-second prompt to run out, Or do you have any error screenshots? Let's examine the causes of early timeouts
So I think maybe Edge runtime is involved. According to https://vercel.com/docs/functions/configuring-functions/duration.
You can't configure a maximum duration for functions using the Edge runtime. They can run indefinitely provided they send an initial response within 25 seconds.
SuiYunsy
commented
1 month ago About api/openai/[...path]: If using edge, as long as the first byte arrives within 25 seconds, the streaming can continue indefinitely. If using nodejs, regardless of when the first byte arrives, the total transmission time will be cut off if it exceeds the max-duration. So the conclusion is to keep api/openai/[...path] as it is——set it as an edge function.
https://github.com/Hk-Gosuto/ChatGPT-Next-Web-LangChain/issues/258#issuecomment-2128771868
SuiYunsy
commented
1 month ago @maxduke Could you please reopen the issue? I think there should be a way to avoid the problem of Edge Functions timing out if they don't receive an API response within 25 seconds. I found the following two links on GitHub, and both approaches set up a keep alive heartbeat in the router:
https://github.com/vercel/ai/issues/487#issuecomment-1847754042
As the docs explain, Edge Functions don't have a maximum streaming time once they started streaming. So a possible workaround for this issue could be to start streaming empty strings with a 2 second interval so that we keep the connection alive while the LLM API call loads.
https://github.com/orgs/vercel/discussions/3553#discussioncomment-7131497
Specifically, apart from sending a response within 25 seconds (which I currently do) you actually have to keep sending messages periodically (unclear at what frequency, but 15 seconds resolves the issue) to keep the stream alive.
But I'm a programming newbie and can't really handle the code😂😂😂 I hope some skilled expert can implement the heartbeat to solve the 25s timeout issue. Thanks a lot!
maxduke
commented
1 month ago @maxduke Could you please reopen the issue? I think there should be a way to avoid the problem of Edge Functions timing out if they don't receive an API response within 25 seconds. I found the following two links on GitHub, and both approaches set up a keep alive heartbeat in the router:
As the docs explain, Edge Functions don't have a maximum streaming time once they started streaming. So a possible workaround for this issue could be to start streaming empty strings with a 2 second interval so that we keep the connection alive while the LLM API call loads.
https://github.com/orgs/vercel/discussions/3553#discussioncomment-7131497
Specifically, apart from sending a response within 25 seconds (which I currently do) you actually have to keep sending messages periodically (unclear at what frequency, but 15 seconds resolves the issue) to keep the stream alive.
But I'm a programming newbie and can't really handle the code😂😂😂 I hope some skilled expert can implement the heartbeat to solve the 25s timeout issue. Thanks a lot!
reopened. Thanks a lot for the information. Let’s see.
Problem Description
You'll get a timeout error if no data is returned from the API after about 25 seconds when deploying the application in Vercel Hobby.
Solution Description
Vercel Functions for Hobby can now run up to 60 seconds. https://vercel.com/changelog/vercel-functions-for-hobby-can-now-run-up-to-60-seconds I'm wondering if the above change can be implemented.
Alternatives Considered
No response
Additional Context
No response