Closed alexklibisz closed 8 years ago
One way to achieve this is with a very simple task that has a start_state
of null
that simply creates three new tasks with the _state
set to fan_out
, push_notification
, and email_notification
respectively. This task can then resolve and remove itself.
The key is to make this as simple and idempotent as possible, perhaps with a single update on the queue, something like this:
var tasksRef = new Firebase('...').child('queue/tasks');
function fanOutTask(originalTask, done) {
var newTasks = {};
['fan_out', 'push_notification', 'email_notification'].forEach(function(taskType) {
var pushId = tasksRef.push().key(); // No network calls, just generates a push ID
newTasks[pushId] = _.assign({
_state: taskType
}, originalTask);
});
tasksRef.update(newTasks, done); // Make sure to use `update` and not `set`
}
@drtriumph Thanks for the great idea! It's essentially a fan out for your fan out!
My team ended up incorporating AWS Lambda functions to handle the processing tasks. So we have a single instance running firebase-queue that proxies the tasks along to a series of lambda endpoints as they are received.
That being said, we probably don't need the solution given by @drtriumph anymore, but I think it will be helpful for anyone else who stumbles upon this problem. I haven't tested the implementation, but it looks very practical so I'm inclined to mark it as resolved.
Is there any way to create a single task and then handle it with multiple distinct workers without chaining specs?
For a simple example, in a messaging app, lets say I want to
Each of these should occur in its own separate node.js process. It doesn't matter what order they occur in.
I know that it's possible to chain specs (i.e. spec_1's finished_state is spec_2's start_state). And I have a simple example working this way.
For my use case, I would potentially have 10 - 12 distinct workers for a single task. My concerns are 1) chaining specs will cause a lot of complexity and 2) if one of the processes is killed or re-started, the following workers will be delayed when they don't have to be b/c they have no dependence on the previous workers.
I have tried the following:
I know it doesn't work this way, but an optimal solution would be as follows:
fan_out
,push_notification
,email_notification
.fan_out
, a queue with specIdpush_notification
, and a queue with specIdemail_notification
.Thanks for building an awesome product!