How to partially fail a batch?

[danilofuchs]

I'm implementing batch support in my application, and I would like to fail only 1 job of the batch, using Promise.allSettled:

    await this.boss.work<PgBossJobData<T>>(
      queueName,
      {
        includeMetadata: true,
        pollingIntervalSeconds,
        batchSize,
      },
      async (jobs) => {
        const results = await Promise.allSettled(
          jobs.map((job) => this.runJob(job, queueName, jobHandler)),
        );

        const errors = results.filter((result) => result.status === "rejected");

        const successes = results.filter((result) => result.status === "fulfilled");
      },
    );

I could use fetch, complete, fail myself, but then I loose all the Worker functionality and have to manage polling myself

Would be nice if work had an API for that, for example:

    await this.boss.work<PgBossJobData<T>>(
      queueName,
      {
        includeMetadata: true,
        pollingIntervalSeconds,
        batchSize,
        autoComplete: false,
      },
      async (jobs, { complete, fail }) => {
        const results = await Promise.allSettled(
          jobs.map((job) => this.runJob(job, queueName, jobHandler)),
        );

        const resultsWithJobId = results
          .map((result, index) => ({ result, jobId: jobs[index].id }))

        for (const {result, jobId} of resultsWithJobId) {
          if (result.status === "rejected") {
            await fail(jobId, result.reason)
          } else {
            await complete(jobId)
          }
        }
      },
    );

work would only poll again if all jobs have been completed or failed

[timgit]

The issue with Promise.allSettled and Promise.all is that it has no concurrency control. v9 and below had support for this via the p-map package and used the teamConcurrency option. However this produced too much complexity for all use cases, since some jobs within a batch may take longer than others, reducing the overall throughput over time.

[danilofuchs]

is that it has no concurrency control

What do you mean by concurrency control in the context of Promise.allSettled? Not sure I understood the requirement

[timgit]

How many promises run in parallel