Open Skn0tt opened 1 year ago
In https://github.com/quirrel-dev/quirrel/discussions/1150#discussioncomment-6278830, @nilooy mentions that it works when switching to quirrel/next-app
.
Hi @nilooy! According to the Next.js docs, switching to runtime = "edge"
changes the API of API Routes completely. You won't be able to use runtime = "edge"
in conjunction with quirrel/next
, which you already found out. Now you're mentioning that with quirrel/next-app
, "streaming doesn't work". Can you go into more detail on that? What exactly doesn't work? Is it related to Quirrel? A reproduction case for that would be lovely.
In #1150 (comment), @nilooy mentions that it works when switching to
quirrel/next-app
.
Thanks for switching it to issue. the error mentioned was gone after switching back to quirrel/next-app
from quirrel/next
but i'm not sure how to make the streaming work.
so, as far i understood, quirrel calls the api back on the scheduled time (or as background task) which is hosted in vercel and has a timeout limit of 60s but i need a long running task. in the code block below, i want to keep this process on until open ai finishes it's response's stream. main goal here is to, bypass the vercel timeout limit of 60s which can be extended to higher number by streaming the response.
export default TestQueue("api/test", async (params) => {
const response = await openai.createChatCompletion({
model: "gpt-3.5-turbo",
stream: true,
messages: [{ role: "user", content: "explain the next js" }],
});
const stream = OpenAIStream(response);
// Respond with the stream
return new StreamingTextResponse(stream);
});
mentioned in vercel stream docs: https://vercel.com/docs/concepts/functions/edge-functions/streaming
Edge Functions must begin sending a response within 30 seconds to fall within the maximum initial response time. Once a reply is made, the function can continue to run. This means that you can use Edge Functions to stream data, and the response will be delivered as soon as the first chunk of data is available.
I don't think that's currently possible with Quirrel. You're returning a StreamingTextResponse
, but Quirrel isn't accessing that return value in any way. It's always returning "OK"
as the response body:
Before we think about solving this, please elaborate on your usecase for this. What's the reason you're accessing OpenAI from a Queue, where you can't send data to your frontend?
the main usecase here is, i'm making a open ai call with quite a large prompt and then parsing the data into json, in total process can take upto 2 min. i can't let my users wait for that long in the frontend. so i must take the path of fan out jobs and let the user know when it's done instead of them keeping in the page for 2 mins.
at the moment there's few talks but not valid solution except defer or a custom one.
some info: https://www.reddit.com/r/nextjs/comments/uhhmga/best_way_to_deal_with_long_background_jobs_when/ https://github.com/vercel/next.js/discussions/34266
Makes sense, thank you!
I don't think that Quirrel currently supports that, and i'd need to think a bit about the best way of implementing support for these long-running jobs. Have you looked into https://www.inngest.com/, would something like that solve your needs?
Ok perfect, inngest has a very different way to solve this issue, but with that, i need to change my entire workflow of next.js api, and will get into strong vendor lock in. I have another way i tested with gcp function, probably will go with that or make a node.js server with bull mq. i will check back later if you have any solutions in mind with quirrel. by the way, Quirrel is really good. Love it.
@Skn0tt can you please rename the title of the issue to Vercel Long Running task (edge function)
which might bring interest from other people who has same needs.
it doesn't allow me to change it.
Discussed in https://github.com/quirrel-dev/quirrel/discussions/1150