Queue worker stops working when _state is indexed

[tsemerad]

Firebase Queue stops working on queues that have a few tasks sitting in the queue, and the worker function is asynchronous, and the tasks node's _state node is indexed. For example, I have the following example queue worker (taken from the README) using firebase version 2.4.0 and firebase-queue version 1.2.1:

var Queue = require('firebase-queue'),
    Firebase = require('firebase');

var ref = new Firebase('https://<your-firebase>.firebaseio.com/queue');
var queue = new Queue(ref, function(data, progress, resolve, reject) {
  // Read and process task data
  console.log(data);

  // Do some work
  progress(50);

  // Finish the task asynchronously
  setTimeout(function() {
    resolve();
  }, 1000);
});

The queue worker's function works fine when a single task is added, but when multiple are added the queue processes one or two of them then stops, although there are still a few in the queue. The same issue happens when 3 or more tasks are already in the queue before the worker is launched.

For example, if I have the following queue set up in my Firebase db:

"queue": {
  "tasks": {
    "keyName": {
      "fileName" : "images.zip"
    },
    "keyName2": {
      "fileName" : "database.db"
    },
    "keyName3": {
      "fileName" : "otherFile.txt"
    },
    "keyName4": {
      "fileName" : "config.xml"
    }
  }
}

With the following security rules:

{ 
  "rules": {
    "queue": {
      ".read": "true",
      "tasks": {
        ".indexOn": "_state",
        ".write": "true"
      }
    }
  }
}

When I start the worker, it will process 2 tasks, then sit idle. Even if I kill and restart the worker process, it won't process any more, as if there are no tasks to process. But it isn't frozen, because if I add a new task to the queue, it'll process 2 of the oldest tasks (not necessarily the most-recently added one), then idle again. After a few minutes of sitting it may eventually process the rest of the queue, but it seems inconsistent.

This behavior is only seen when the worker's callback function does something asynchronous. If I comment out the progress() or setTimeout() functions above, it'll churn through any number of tasks (I tested about 70 at once) without hanging.

Also, this behavior appears to be caused by having an index on the queue. If I remove ".indexOn": "_state" from the queue's security rules, the worker immediately continues processing the remaining items in the queue.

For now I've removed the index, and everything works fine. Except for the fact that I'm always warned that I should have an index on my queue whenever I launch the application.

[cbraynor]

Can you try again - yesterday we deployed a patch to the realtime database to fix what we think was causing this issue (this repro was increadibly helpful, thank you). Things should be working correctly now

[tsemerad]

The issue is still there. I uncommented my ".indexOn": "_state", rule, saved, then filled the queue with about 70 tasks and it processed 3 of them. Commenting out the index once again resulted in clearing out the rest of the queue.

I was thinking - is it a problem that the default spec for a queue has a default _state of null? I imagine that trying to index tasks by _state, where most tasks don't have a _state value, wouldn't make for a useful index. For my purposes, the queue is used fairly infrequently so the index is not a priority. The main issue is that the Firebase Queue library warns you if you don't have it indexed, so you index it thinking it can't hurt, but then it does. Maybe the library shouldn't warn users to index if they're using the default spec?

[maxrevilo]

firebase-queue used to work perfectly with _state on null, but now it is failing, I don't know why. I have the same problem: https://github.com/firebase/firebase-queue/issues/42. As a work arround you can use a custom spect with start_state set to 'queued' for example, it should work perfectly.

[alexdrel]

We have exactly the same issue - only one item is picked up from queue when the worker is busy. Workaround by commenting out ".indexOn": "_state" helps. @tsemerad - thanks!

[cbraynor]

The database was updated again on Friday last week which fixed the underlying issue. If you're still having issues, please reopen

[simenbrekken]

I've tried both with and without the index, but my queues are still going idle after a while.

Here's my worker implementation:

const notificationsRef = ref.child('notifications');
notificationsRef.child('specs').set({
  format: {
    in_progress_state: 'formatting',
    finished_state: 'formatted',
  },

  send: {
    start_state: 'formatted',
    in_progress_state: 'sending',
  },
});

const queues = [
  new FirebaseQueue(notificationsRef, { specId: 'format' }, format),
  new FirebaseQueue(notificationsRef, { specId: 'send' }, send),
];

After a while none of the queues are picking up tasks, they simply sit there unmodified since when they were first pushed.

The worker is running on Heroku in a hobby (always on) dyno. I've also started logging .info/connected but I'm not seeing anything that would indicate the connection being dropped.

[cbraynor]

I think this is separate - if either the format or send function don't call resolve or reject in every scenario, the worker might get stuck in a busy state and not ever pull new tasks off the queue

[simenbrekken]

I was under the impression that the task would timeout and thus reject after 5 minutes even if a worker failed to resolve/reject the current task?

[cbraynor]

The task will timeout and another worker will take the original task, but the original worker will not take on any more jobs until the current processingFunction either resolves or rejects

[simenbrekken]

I see. Guess I'll have to take a closer look at what not being resolved/rejected. Is there any built-in debugging I can use?

[ur-technology]

I'm having the same problem: Only the first queue item is processed when I use "tasks": { ".indexOn": "_state"} If I remove the index, the problem goes away. I am using version 1.5.0. Is there any work around to this?

[katowulf]

@urcapital Since this issue has been closed for over 4 months, please start by creating a new issue. Feel free to link to this one for context if you feel they are the same problem.

Be sure to include a minimal repro of the problem or failing test case. You will also want to include complete version info (which version of the SDK were you able to reproduce this against? What happens if you update to the latest?), sample data, and steps to run your example.