Twitter event repeatedly causes locker to puke and die

[othiym23]

Saw the following error in a locker, restored it from last good backup, waited an hour or so, and got the exact same error on the exact same event:

25 Feb 21:37:04 - verbose: received event for contact://twitter/contact?id=twitter#21237045 action(update)
25 Feb 21:37:04 - error: [TypeError: Object 1 has no method 'apply']
25 Feb 21:37:04 - error: 'TypeError: Object 1 has no method \'apply\'\n    at /home/node_modules/async/lib/async.js:609:43\n    at Collection.remove (/home/node_modules/mongodb/lib/mongodb/collection.js:173:12)\n    at removeCurrent (/home/Common/node/ldatastore.js:157:11)\n    at /home/Common/node/ldatastore.js:83:9\n    at Object.wrapper [as oncomplete] (fs.js:304:17)'

25 Feb 21:37:04 - info: Shutting down...
25 Feb 21:37:04 - info: Signalling photos to shut down (pid 172)
25 Feb 21:37:04 - info: Signalling places to shut down (pid 170)
25 Feb 21:37:04 - info: Signalling links to shut down (pid 184)
25 Feb 21:37:04 - info: Signalling search to shut down (pid 182)
25 Feb 21:37:04 - info: Signalling contacts to shut down (pid 219)
25 Feb 21:37:04 - info: Waiting for photos to exit.
warn: Forever detected script exited with code: 1
warn: Forever restarting script for 1 time
The "sys" module is now called "util". It should have a similar interface.

[mdz]

It looks like it's getting an uncaught exception during shutdown

[othiym23]

If the following function fails due to something coming from Mongo, it's not clear to me where the callback with the error information will be handled, but it appears to be propagated out of ldatastore.js

function getMongo(owner, type, callback) {
    var m = colls[owner][type];
    if(!m) {
        try {
            mongo[owner].addCollection(owner, type);
        } catch (E) {
            return callback(E, []);
        }
        m = colls[owner][type];
    }
    return m;
}

[mdz]

https://github.com/LockerProject/Locker/pull/876 should make it easier to debug uncaught exceptions during shutdown

[smurthas]

That is definitely a bug - that function should just throw the error as it is sync and the async functions above it should try catch.

Beyond that, looks like a bug in ldatastore or lsyncmanager. Something is passing an Object instead of a function as the callback to removeCurrent in ldatastore.

I'm going to fig the first bug, put in some debug code and let it run.

[smurthas]

As best I can tell, when building the async queue in either the deleteData or addData functions of ldatastore, something goes wrong and results in a callback being set for an individual queue task that isn't actually a function so when async tries to call apply it can't. I wrapped the queue's process function to print a debug statement if it catches an error. The debug statement will look like this:

ERROR: caught error while processing q on task <task with bad callback>

It won't bring the whole ship down now, but we should take the try catch out once we understand and fix this.

[othiym23]

Closing as abandoned.