handling file upload in request handler vs middleware issues

[2fours]

I'm trying to upload files to rackspace through my node server without saving the file to disk. If I use express.bodyParser() in the middleware chain like so (coffeescript)

 app.use express.compress()
 app.use express.bodyParser()   
 app.use express.session(
  secret: sessionSecret
  store: sessionStore
 )
app.use passport.initialize()
app.use passport.session()
app.use app.router

The fileupload works fine (minus the disk saving), I can see the files have been parsed from the form in the request.

If I then omit body parser from the chain (and use json and urlencoded middlewares) and attempt to process the form in the request handler by overriding the form.onPart function, problems arise. At first I thought it was due to this issue as I'm using passport: https://github.com/jaredhanson/passport/pull/106 but I've implemented those changes and the problem still persists.

I've noticed that in incoming_form.js in the write function the code errors because the buffer at this point is not in human readable form, whereas when the form is processed in the middleware chain before reaching the request handler the buffer is human readable so somewhere between those two points the buffer is being changed in this way.

I'm using SSL, gzip compression, formidable 1.0.11, and node 0.8.22

Any help greatly appreciated,

Adam

[2fours]

Here is the stack trace for the error, I've highlighted the relative portion where the write fails because it can't parse the buffer:

2013-04-19T11:56:21.896Z - error: uncaughtException date=Fri Apr 19 2013 05:56:21 GMT-0600 (MDT), pid=20820, uid=1000, gid=100, cwd=/home/adam/dev/surespot/web/server, execPath=/usr/bin/node, version=v0.8.22, argv=[/usr/bin/node, /home/adam/dev/surespot/web/server/cluster.js], rss=67649536, heapTotal=61274880, heapUsed=31876680, loadavg=[1.14306640625, 1.1328125, 1.13916015625], uptime=484444.285647342, trace=[column=17, file=/home/adam/dev/surespot/web/node_modules/formidable/lib/incoming_form.js, function=IncomingForm.write, line=145, method=write, native=false, column=12, file=/home/adam/dev/surespot/web/node_modules/formidable/lib/incoming_form.js, function=, line=95, method=null, native=false, column=17, file=events.js, function=IncomingMessage.EventEmitter.emit, line=96, method=EventEmitter.emit, native=false, column=10, file=http.js, function=IncomingMessage._emitData, line=359, method=_emitData, native=false, column=21, file=[as onBody] (http.js, function=HTTPParser.parserOnBody, line=123, method=parserOnBody, native=false, column=22, file=http.js, function=CleartextStream.socket.ondata, line=1825, method=socket.ondata, native=false, column=27, file=tls.js, function=CleartextStream.CryptoStream._push, line=544, method=CryptoStream._push, native=false, column=20, file=tls.js, function=SecurePair.cycle, line=898, method=cycle, native=false, column=13, file=tls.js, function=EncryptedStream.CryptoStream.write, line=285, method=CryptoStream.write, native=false, column=26, file=stream.js, function=Socket.ondata, line=38, method=ondata, native=false],

stack=[Error: parser error, 0 of 2048 bytes parsed,     at IncomingForm.write      (/home/adam/dev/surespot/web/node_modules/formidable/lib/incoming_form.js:145:17),     at 

IncomingMessage. (/home/adam/dev/surespot/web/node_modules/formidable/lib/incoming_form.js:95:12), at IncomingMessage.EventEmitter.emit (events.js:96:17), at IncomingMessage._emitData (http.js:359:10), at HTTPParser.parserOnBody as onBody, at CleartextStream.socket.ondata (http.js:1825:22), at CleartextStream.CryptoStream._push (tls.js:544:27), at SecurePair.cycle (tls.js:898:20), at EncryptedStream.CryptoStream.write (tls.js:285:13), at Socket.ondata (stream.js:38:26)]

[2fours]

Figured out the problem. My upload image request handler looks like:

app.post "/images/:fromversion/:username/:toversion", ensureAuthenticated, validateUsernameExists, validateAreFriends, (req, res, next) ->

so I have some custom middlewares where I'm doing some validation. These validation middlewares talk to the database and apparently they were suffering from the same problem as the passport middleware. It seems that pausing and resuming the requests in my middlewares fixes the issue.

[schteppe]

I recently got the same problem, found this and could fix it easily. Thanks! A comment I would like to add is that there's a middleware library for doing this kind of pause. See https://github.com/randymized/pause-request-body