Get current job count in the queue.

[sayeed205]

Hey there I tried out the package and it working as expected.

I have some use case for how many current jobs are there in a specified queue. for example I have a job name article-processor.

and I always send an id in the payload. and I need to retrieve the job count (both processed and queued) and send a socket event when there is no job queued with the id I sent in the payload.

Can you please expose an API so that I can retrieve the count.

[shiny]

Oh great idea, let me think about it.

[shiny]

For pending jobs

const queue = await app.container.make('queue')
const pendingJobsTotal = await queue.length(<YourQueueName>)

this variable queue is a node-resque Queue, you can find the API here https://node-resque.actionherojs.com/classes/Queue.html

for the pending job count of the specified queue, you can get the number from await queue.length(<YourQueueName>)

[!NOTE] under the hood, it is a redis set in key resque:queue:<YourQueueName> with value {"class":"ArticleProcessor","queue":"<YourQueueName>","args":[298868]}

If you don't like to enqueue the same job with same args, you can use Plugin.queueLock to avoid.

May be we can design an API here to count in job level, but i can imagine that could be slow if there is a lots of pending jobs.

For processed (failed) jobs

you can get the total number of all processed(and failed) jobs by queue.stats().

[!NOTE] it is the redis key: resque:stat:processed and resque:stat:failed with a number value

But there is no API to get the number of processed jobs in specific queues, not even distinguish different jobs. Just a total number of all, as Resque designed that.

Since redis is a in-memory kv database, the cost is rare high. So details of processed jobs won't keep stay in redis, you should not rely on it. Only failed details stays.

[sayeed205]

Hey thanks for the detailed explanation.

I think my use case can be satisfied by what you have explained and guided.

btw can I start the job listener in different process ? like rn the process starts listening for jobs right after the server starts.

I have 2 or more jobs and I want them to be in separate terminal/process. for example - I have two jobs rn spider and articleProcessor spider - crawls links from a domain. gets called from an api end point articleProcessor - gets article from the crawled links found in spider job. gets called from spider job

why I needed in different process?

I am using crawlee to crawl the sites. and facing some issues while using in same process. like it skips some of the links. I tried in separate process using rlanz/bull-queue and it was working perfectly. but if I run it in same process it was skipping some links.

[shiny]

Sure. You can define sipder & articleProcessor in two different queues, e.g.

class SipderJob extends BaseJob {
    queueName = 'spider'
    async perform() {
        // discover links
        await ArticleProcessorJob.enqueue(id)
    }
}

class ArticleProcessorJob extends BaseJob {
    queueName = 'ArticleProcessor'
    async perform(id: number) {
        // get article
    }
}

The option runWorkerInWebEnv in config/resque.ts is default true. That makes adonis web server start the workers, turn it off first.

Then run workers in separated processes by specifing --queue-name in resque:start.

Start a multiwoker, only process the job in spider queue

node ace resque:start --schedule=false --worker --is-multi --queue-name=spider

Only for ArticleProcessor queue

node ace resque:start --schedule=false --worker --is-multi --queue-name=ArticleProcessor

Start a scheduler

node ace resque:start --schedule --worker=false

Since resque is for distributed jobs, you can start processes as many as you like, even in different machines. that is fine.

[shiny]

--queue-name is an array flag. listen on default and spider queue both

node ace resque:start --schedule=false --worker --is-multi --queue-name=spider --queue-name=default

[!NOTE] if you omit queueName on Job, it is default.

[sayeed205]

Thanks mate. everything working fine according to my use cases.

Closing this issue now.

[shiny]

Glad to see that