search

marcello3d / node-mongolian

[project inactive] Mongolian DeadBeef is an awesome Mongo DB driver for node.js
https://groups.google.com/group/node-mongolian
zlib License
350 stars 50 forks source link

Error: Buffer is not large enough #111

Open qharlie opened 11 years ago

qharlie commented 11 years ago

I'm getting this error a lot , here is the full stack trace:

Error: Buffer is not large enough. Expected 1745879154, have 36 new Response (/opt/deploy/formhog/node_modules/mongolian/node_modules/buffalo/lib/mongo.js:255:15),Connection. (/opt/deploy/formhog/node_modules/mongolian/lib/server.js:227:28),Connection.EventEmitter.emit (events.js:88:17),Socket. (/opt/deploy/formhog/node_modules/mongolian/lib/connection.js:61:22),Socket.EventEmitter.emit (events.js:88:17),TCP.onread (net.js:397:14)

marcello3d commented 11 years ago

I have seen this occasionally as a cascade from a program error/exception. Can you reproduce it?

qharlie commented 11 years ago

Ahh thats probably it. Let me try something.

awwright commented 11 years ago

Yeah I have this problem too, though usually only when I haven't properly handled an error somewhere else. I haven't been able to reliably reproduce it, it just sort of shows up among a bazillion other errors when things go wrong.

awwright commented 11 years ago

I've only been able to reproduce it after another error. I'm suspecting there may be a missing var or similar, somewhere. But I can't see anything obviously wrong with the code!

2013-03-14T15:41:51.285Z - Uncaught Exception: Error: Mongo query should have updated 1 documents, but only updated 0
    at lib/transform.HTTPAuto_typeMongoDB_Post.js:178:13
    at mongolian/lib/collection.js:97:13
    at mongolian/lib/util.js:19:13
    at mongolian/lib/db.js:69:25
    at mongolian/lib/util.js:19:13
    at MongolianCursor.next (mongolian/lib/cursor.js:167:9)
    at mongolian/lib/cursor.js:171:18
    at mongolian/lib/util.js:19:13
    at mongolian/lib/cursor.js:125:9
    at mongolian/lib/util.js:19:13
2013-03-14T15:42:02.393Z - Uncaught Exception: Error: Buffer is not large enough. Expected 1159, have 101
    at new Response (buffalo/lib/mongo.js:255:15)
    at Connection.<anonymous> (mongolian/lib/server.js:227:28)
    at Connection.EventEmitter.emit (events.js:94:17)
    at Socket.<anonymous> (mongolian/lib/connection.js:61:22)
    at Socket.EventEmitter.emit (events.js:94:17)
    at Socket.<anonymous> (_stream_readable.js:648:5)
    at Socket.EventEmitter.emit (events.js:91:17)
    at emitReadable (_stream_readable.js:358:10)
    at onread (_stream_readable.js:343:7)
    at ReadableState.onread (_stream_readable.js:64:5)
2013-03-14T15:42:04.999Z - Uncaught Exception: Error: Buffer is not large enough. Expected 1701080942, have 101
    at new Response (buffalo/lib/mongo.js:255:15)
    at Connection.<anonymous> (mongolian/lib/server.js:227:28)
    at Connection.EventEmitter.emit (events.js:94:17)
    at Socket.<anonymous> (mongolian/lib/connection.js:61:22)
    at Socket.EventEmitter.emit (events.js:94:17)
    at Socket.<anonymous> (_stream_readable.js:648:5)
    at Socket.EventEmitter.emit (events.js:91:17)
    at emitReadable (_stream_readable.js:358:10)
    at onread (_stream_readable.js:343:7)
    at ReadableState.onread (_stream_readable.js:64:5)
awwright commented 11 years ago

If only half a message of the stream were to come in (which is perfectly legal), what would happen here? https://github.com/marcello3d/node-mongolian/blob/f1a6bdc547db1021321acc869d530ac05dda24d3/lib/connection.js#L59-L79

It appears that there's no test to make sure we have a complete message before emitting a "have packet" event.

marcello3d commented 11 years ago

The while loop condition should handle that case.

awwright commented 11 years ago

So this probably is the problem then?

I'm currently re-writing the file to avoid Buffer re-allocations, among other things

marcello3d commented 11 years ago

The solution might be to move the emitting on line 61 to the bottom of that block. That way if the message handler throws an exception, the buffer is in a consistent state.

awwright commented 11 years ago

That might be the case for the second and beyond errors ("Expected 1701080942") but the first one seems to simply be an incomplete packet, with no prior error having been thrown.

Oh wait...

awwright commented 11 years ago

Yeah, that's exactly the problem: whenever an Exception is thrown in an emitter hook, it interrupts the stream handling and an entire segment goes unparsed. So, how should errors be handled? Hmmm

awwright commented 11 years ago

Should it really be disconnecting on error by default: https://github.com/marcello3d/node-mongolian/blob/f1a6bdc547db1021321acc869d530ac05dda24d3/lib/server.js#L212-L215 ?

Here's my file that seems to fix: https://gist.github.com/Acubed/5167758 And available in commit Acubed/node-mongolian@2b48848c31cc80c4d0b18483cd401eeea1b9033d

awwright commented 11 years ago

Perhaps the event should be called with process.nextTick? That would isolate any errors from affecting the main loop. I'm not so sure if it should disconnect from the database, but may be that too.