Queue metrics says the waiting jobs count is 0

[farhadnowzari]

Describe the bug We have a scheduled jobs which will check the database for records which need some calculation and if there is any it will queue a job for them. We are queueing 11k jobs and then when this scheduled job runs again we check if there is any job in queue which is running or waiting we will not queue the jobs again until the queue is empty.

The metrics says there is one jobs running and NONE waiting which is a bit strange.

Affected Coravel Feature Queuing

Expected behaviour It is expected that the Waiting count be the jobs in the queue except the one which is waiting.

[jamesmh]

How often is your queue consuming the queued tasks (this is configurable via appsettings and the default is every 30 seconds if it hasn't been changed)?

[farhadnowzari]

we are using the default value.

ah, so this waiting is not the count of all the tasks in the queue. because we though the WaitingCount means the total number of tasks that are in queue and waiting to be processed. if this is the case how can we get the total number of tasks which are queued?

The waiting tasks is the count of the next task which will be popped for invoke?

[jamesmh]

Yes, so "running" is the count of tasks that are currently being processed right now.

"waiting" are tasks that are being consumed at the moment but waiting to do the actual work.

So to get the count of all tasks, including those that are in the queue and have not been "activated" so to speak, it brings up some questions:

• Does that count include the tasks currently running?
• Does that count include tasks who are about to be processed?

I think it would be correct to basically create another property on the metrics where the count would be all tasks who have not yet started any work on a thread yet.

Is that essentially what you would be looking for?

[jamesmh]

Also, looking at the docs I kinda suspect that the current behaviour might be "wrong. The docs say:

WaitingCount(): The number of tasks waiting to be executed.

I think you're correct in thinking that this number includes all tasks that are both (a) in the queue but haven't been touched at all and (b) tasks that have been taken out of the queue but not processing yet.

These points, (a) + (b) are an implementation details and so I think right now I'd consider the current behaviour more like a bug.

I'll let you know when I get this changed 👍

[farhadnowzari]

Thank you very much. I also checked the code in the repo before our comments here I realized that yeah the waiting is the one thats going to be poped for processing. I think we can have another property to get all the ones which are not waiting to be processed or are not running. That would solve it for us :) Thanks again

[jamesmh]

I made some minor tweaks to this - it's available in 4.0.3.

In the end, once I dug in, the code seemed to be correct... I've added a test to make sure that it works as expected.

Give 4.0.3 a try and let me know if there are any other issues 👍

[farhadnowzari]

Thank you very much I accidentally closed the issue here 😅. I will check this out today and will let you know :) .

I would be happy to help with these changes too whenever I have time. Coravel is a really fluent library, I like it. It reminds me of Laravel jobs ^^. There are some questions would love to discuss them like having jobs queued in different queues with different IDs and assign workers to those queues. Because we needed this but for now we handled it our selves. Also invoking a single job without calling its invoke method because it is nicer if the engine can asynchronously run single jobs.

quick question though. did you add a new property or you changed the already existing ones?

[farhadnowzari]

Just checked it out the WaitingCount is now actually giving the number of jobs in queue correctly 👍 thanks 😊

[farhadnowzari]

The first time that I ran it, the number was right but after resetting the dotnet webapi the waiting count is zero again. I will investigate it more.