Better handling of malformed queue items

[m-tse]

Yesterday, I purposely created a malformed queue item, hoping it would be ignored, for the purpose of keeping the queue subtree visible within the Firebase interface. Otherwise, the queue subtree would disappear, and I'd have to manually recreate it (just 1 extra annoying step) if I wanted to manually add any new queue entries. It looked like this:

{
"backups": {
  "backup-metadata": {
  ...
  },
  "jobs": {
  ...
  },
  "queue": {
    "placeholder": "placeholder"
  }
}

I don't have the exact error, but it should be easy to reproduce. What basically happened was all my workers kept trying to process this malformed queue entry over and over again, and failed, causing CPU/network/disk space usage to spike up. Eventually, the workers started dying off (not at the same time, they died off hours apart from each other over the course of 24 hours), presumably due to some race condition. I was notified of this error when all the workers died, and the nagios check warned me.

In the event of malformed entries being put into the queue, I think we should handle them a bit more gracefully.

[cbraynor]

I agree we should handle malformed (non-object) entries on the queue better, I'll fix that.

For this specific use-case, I think most people will be appending to the queue in a programmatic way (and not through the data dashboard) so it shouldn't matter whether the queue node 'exists', besides you could always go to https://yourfirebase.firebaseio.com/queue in the browser to not have to re-enter it each time

[m-tse]

Right, haha this issue was definitely for the case of "handle malformed queue entries better", and not for "show the queue subtree", which was just my laziness that discovered this issue.

[cbraynor]

Fix upcoming in #6

[cbraynor]

Fixed