Handling tasks multiple times without chaining specs?

[alexklibisz]

Is there any way to create a single task and then handle it with multiple distinct workers without chaining specs?

For a simple example, in a messaging app, lets say I want to

• Fan out the message
• Send a push notification to a user's device with the message.
• Send an email to the user with the message.

Each of these should occur in its own separate node.js process. It doesn't matter what order they occur in.

I know that it's possible to chain specs (i.e. spec_1's finished_state is spec_2's start_state). And I have a simple example working this way.

For my use case, I would potentially have 10 - 12 distinct workers for a single task. My concerns are 1) chaining specs will cause a lot of complexity and 2) if one of the processes is killed or re-started, the following workers will be delayed when they don't have to be b/c they have no dependence on the previous workers.

I have tried the following:

• Chaining specs. It works, but like I said it's sub-optimal for my use case.
• Creating multiple specs with starting_state set to null. Then running a separate process for each spec (i.e. process 1 takes spec_1, process 2 takes spec_2). Still, only one of the processes picks up the task, and it's now random.

I know it doesn't work this way, but an optimal solution would be as follows:

• have a spec for each process: fan_out, push_notification, email_notification.
• define a queue with specId fan_out, a queue with specId push_notification, and a queue with specId email_notification.
• any time a task is pushed, all three queues can handle the task.

Thanks for building an awesome product!

[cbraynor]

One way to achieve this is with a very simple task that has a start_state of null that simply creates three new tasks with the _state set to fan_out, push_notification, and email_notification respectively. This task can then resolve and remove itself.

The key is to make this as simple and idempotent as possible, perhaps with a single update on the queue, something like this:

var tasksRef = new Firebase('...').child('queue/tasks');
function fanOutTask(originalTask, done) {
  var newTasks = {};
  ['fan_out', 'push_notification', 'email_notification'].forEach(function(taskType) {
    var pushId = tasksRef.push().key(); // No network calls, just generates a push ID
    newTasks[pushId] = _.assign({
      _state: taskType
    }, originalTask);
  });
  tasksRef.update(newTasks, done); // Make sure to use `update` and not `set`
}

[alexklibisz]

@drtriumph Thanks for the great idea! It's essentially a fan out for your fan out!

My team ended up incorporating AWS Lambda functions to handle the processing tasks. So we have a single instance running firebase-queue that proxies the tasks along to a series of lambda endpoints as they are received.

That being said, we probably don't need the solution given by @drtriumph anymore, but I think it will be helpful for anyone else who stumbles upon this problem. I haven't tested the implementation, but it looks very practical so I'm inclined to mark it as resolved.