FirebaseExtended / firebase-queue

MIT License
786 stars 108 forks source link

Enhancement - Time delay, process after time, or peek, function for queue. #35

Open uofmmike opened 8 years ago

uofmmike commented 8 years ago

Would love to see a feature that would allow a "process after" setting. This would be similar to the "peek" issue https://github.com/firebase/firebase-queue/issues/28 that asks for some sort of pre-check to see if it can be fired off to a worker.

@drtriumph (Chris) I came across your other post in the google group, and i'd have to say i'm VERY interested in this! Any idea where on the feature bored this one may fall???? (https://groups.google.com/forum/#!msg/firebase-talk/suJTZXG3CgU/wNjfw3ZnCQAJ)

I'm a big fan of the queue, its an awesome addition to the firebase stack!

cbraynor commented 8 years ago

Your vote has very much been counted. For now the work-around for a delayed job is a 'waiting room' style task that claims a job, resolves it if the condition is met, or delays before rejecting it if not - with a high enough number of retries it should achieve what you need, and you can customize the granularity by changing the delay / number of retries.

I can't promise any changes in the near-term as things are pretty busy here at Firebase HQ, but I do accept PRs ;-)

uofmmike commented 8 years ago

Response appreciated, i'll give it a look!

soaxelbrooke commented 8 years ago

This does sound like a great feature - originally I was considering using something like SQS for this, but thought, "wait, shouldn't firebase be able to do this?"

doowb commented 8 years ago

I created firebase-cron to try to solve this. I had been looking for a way to do job scheduling and I think this matches up with firebase-queue pretty well.

I first tried to see if there was a good way to incorporate the cron jobs into firebase-queue so I could just do a PR, but since this is requiring polling the data (due to timestamps changing), I didn't find a suitable place to put this. A separate library also keeps the concerns separate and allows running just the cron scheduling on a server by itself, while still scaling out queue workers.

I'd appreciate any feedback, bug reports, and/or PRs

henry74 commented 8 years ago

@doowb Thanks for doing this. I've been looking for something like this. Going to take a peek at your examples to understand how to use it. Will it only push data onto a queue at a specific time or does it also allow running some code at a specific time as well?

doowb commented 8 years ago

It only pushes data into a queue at a specific time (based on the cron pattern). The idea is that you would setup your queue code to run the job when the data is added. This lets you push in any data object and use the same conventions as firebase-queue without firebase-cron needing to know anything about the data.

One feature I'd like to add is a run once method that just takes a time in the future and runs once and removes the object so it doesn't run again. I haven't implemented this yet, but it would be something that would solve some of the use cases discussed here.

RdeWilde commented 7 years ago

I would like to see this feature integrated. Scheduling, delaying, timeouts when workers becomes unresponsive, deadlines. I understand a worker could handle this with custom code, but it's much nicer to have one solution we can all use for such a general feature.

vekexasia commented 7 years ago

Couldnt we define a spec of a given timeout/2 &retry=1 and let the worker do nothing? so that the next spec starts from the failure of this?

katowulf commented 7 years ago

See also this discussion on Google Groups; covers using a tiny worker to query based on timestamp and inject items into the queue at scheduled times.

funkju commented 7 years ago

@drtriumph Regarding your workaround, does a delay then reject() + retries make the queue not FIFO? Does a rejected task get put at the bottom of the queue?