Running as background jobs with Qstash

[nilooy]

i am trying to use vercel edge function's streaming with ai sdk for long running job(up to 120s) with https://docs.upstash.com/qstash

it works perfectly locally but when deployed on vercel, it doesn't. it just stream for few milli seconds and stops. (no error) i'm not sure exactly where the issue is. At first glance seems like, the issue is on qstash but then it works locally and not in vercel, so it indicates that the issue might be there.

i would love to know any feedback about this approach for fan out jobs with edge function's streaming.

calling this to run queue:

import { baseUrl, qstashClient } from "@/lib/queue/qstash";

export default async function handler(req, res) {
  const job = await qstashClient.publishJSON({
    url: baseUrl + "/api/test/chat",
    // or topic: "the name or id of a topic"
    body: {},
    retries: 0,
  });

  return res.status(200).json({ ok: true });
}

the queue that run as background job

import { Configuration, OpenAIApi } from "openai-edge";
import { OpenAIStream, StreamingTextResponse } from "ai";
import { validateSignatureEdge } from "@/lib/queue/qstash";

// Create an OpenAI API client (that's edge friendly!)
const config = new Configuration({
  organization: process.env.OPENAI_ORG,
  apiKey: process.env.OPENAI_API_KEY,
});

const openai = new OpenAIApi(config);

export const runtime = "edge";

async function handler(req: Request) {
  const response = await openai.createChatCompletion({
    model: "gpt-3.5-turbo",
    stream: true,
    messages: [{ role: "user", content: "how next.js works?" }],
  });
  const stream = OpenAIStream(response, {
    onToken: (token) => {
      console.log({ token });
    },
    onStart: () => {
      console.log("started");
    },
  });
  // Respond with the stream
  return new StreamingTextResponse(stream);
}

// it's a custom signature validator of qstash
export default validateSignatureEdge(handler);

[MaxLeiter]

Sorry, I don't see how this could be an AI SDK problem if it works fine without QStash. Perhaps it works locally because they detect you're in dev and handle things differently?

[inducingchaos]

@nilooy did you manage to figure this out? I want to do something similar.

I'm using the AI SDK with OpenAI's Chat Completion to interface with SMS via Twilio, and need to accommodate for longer runs than 25s.

One solution as mentioned in Vercel's docs is streaming (to keep the Edge function alive), however since this is happening server-side to respond to SMS messages, it's sort of a "headless" environment, and I have no where to stream to — and streaming from Edge seems to be the only solution if I want to stay within the Next.js environment.

My thought was to use Upstash's QStash with callbacks and open a stream there, but I haven't tried to implement this yet. Maybe I'm missing something.

[nilooy]

@rileybarabash It was such a pain as I was developing as indie dev to find a good solution for this. I am currently using Google cloud function and cloud task to achieve the background task with next.js but it's bit time consuming to setup. There's one paid(also free plan) solution called defer.run that integrates easily with next.js and vercel and it seems to be working pretty well for my colleagues. It has quite a long timeout but if you have too many concurrent task, it may get expensive. You may look at their pricing. I have tried quirrel and qstash both, none works with streaming and technically I understand the limitations. There's also this trending library called trigger.dev which seems nice and their long running jobs are in WIP. So may be not an option for now. Also netlify has long running jobs now but not for free plan also vercel now has timeout settings up to 5 min in pro plan. You have to decide on your budget, time and needs which one to use. I suggest to give defer.run a try.

[inducingchaos]

Thank you for the thorough response! Going to look into defer.run.

I also wasn't aware that Vercel offered longer serverless timeouts on their Pro plan, that may be a decent option as well.

[trompx]

Looks like defer.run is not an option anymore unfortunately.

Defer is reaching the end of its service life on May 1, 2024

[inducingchaos]

I actually got Defer running pretty well, it was easy to set up and manage. Very unfortunate to see them sunset.

The owner suggested migrating to trigger.dev. Join their Discord server, the answers you need should be there.