seomoz / qless-core

Core Lua Scripts for qless
MIT License
85 stars 33 forks source link

Do we plan to have function to delete the queue. #54

Open xmywang opened 10 years ago

xmywang commented 10 years ago

Hi!, We need dynamic to create the queue and delete the queue. Are we going to support this function? If not, any instruction that we can do it by our own lua script?

More precisely, we are trying to solve the problem that some jobs in the queue taking very long time and some jobs in the queue taking short time. Here is the situation: Our jobs are grouped in certain way that we do not want each group of jobs interacts each other. The group is dynamically created and can be gone forever. Currently, we are trying to dynamically associate each group to its own queue. However, the number of groups is not fixed. So if the groups are created and then gone forever, we have to delete the those queues to avoid leaking.

Thanks a lot!

dlecocq commented 10 years ago

This should be possible by 1) ensuring that all the jobs in that queue are completed or canceled and then 2) calling queue.forget on the queue to remove it from the list of known queues. This will avoid all leakage.

xmywang commented 10 years ago

For our case, the deletion of queue is triggered by the user from our UI. We need cancel all jobs in the queue asap. So we write a small lua script: In queue.lua, add this function: {code} -- List jobs and their depends ids function QlessQueue.jobs(now, name, count)

local queue = Qless.queue(name) -- These are the ids that we're going to return. We'll begin with any jobs -- that have lost their locks local jids = queue.work.peek(count) table.extend(jids, queue.locks.peek(now, 0, count - #jids)) table.extend(jids, queue.scheduled.peek(now, 0, count - #jids)) table.extend(jids, queue.recurring.peek(now, 0, count - #jids))

local dependents = {} for _, jid in ipairs(jids) do dependents[jid] = redis.call( 'smembers', QlessJob.ns .. jid .. '-dependents') or {} end table.extend(jids, dependents) return jids end {code}

In api.lua, we write this

{code} QlessAPI['job.batch_cancel'] = function(now, queue, count) local jids = QlessQueue.jobs(now, queue, count)
Qless.cancel(unpack(jids)) return #jids end {code}

In this way, our application can repeat calling job.batch_cancel to cancel all jobs in the queue. Then we can safely delete the queue.

Let's know if this is OK or not.

Thanks a lot!

Yan