timgit / pg-boss

Queueing jobs in Postgres from Node.js like a boss
MIT License
2.16k stars 160 forks source link

Connection pooling #289

Closed arslnb closed 2 years ago

arslnb commented 2 years ago

I have a node app where I call this function fomr our main app.ts upon starting – 

export async function startCron(): Promise<void> {
  const boss = new PgBoss({
    connectionString: process.env.DATABASE_URL,
    schema: "tasks",
  });
  boss.on("error", (error) => console.error(error));
  await boss.start();

  boss.schedule(
    "heartbeat",
    "0 1 * * *",
    { isHb: true },
    { tz: "America/Chicago" }
  );

  boss.subscribe("heartbeat", executeHeartbeat);
  boss.subscribe("doc-update", executeStoreUpdate);
  boss.subscribe("element-update", executeItemUpdate);

  boss.onComplete("doc-update", (job) => {
    checkAndAutosend(job.data.request.data.docId);
  });
}

Basically, once a day I run executeHeartbeat which searches for all "stores" that need updating, and then queue a "executeStoraUpdate" job, which inturn searches for all items in that store that need updating and queue a "executeItemUpdate" job, which calls a third party service and update the price in database.

My understanding is that the default number of connections PgBoss makes to the postgres backend is 10. However, I use render.com have hit the 100 connection limit pretty much every other day. Upon investigating (I ran SELECT * FROM pg_stat_activity;), I found that pg-boss had 20-30 idle connections at any given point.

I'm using pg-boss version 5.2.3 if it helps.

timgit commented 2 years ago

I haven't run an app in render before, but I would guess that it's doing some autoscaling for you and each instance will have its own connection pool.