File uploading being cancelled on some files

[tegandbiscuits]

I'm trying to use Skipper to save uploaded images, but it's only working on some files. Some files finish, and successfully write to the disk. Some files finish uploading, but the file that's written is empty, and some files just hang on uploading until it times out. When it cancels or hangs, it's on the same file, so it isn't randomly not working.

I found two working and non-working files to log the callback, and this is what it said (these weren't uploaded all at once, and the working-not-working pattern isn't a coincidence)

[ { size: 2005363,
    type: 'image/jpeg',
    filename: 'Success-1.jpg',
    status: 'bufferingOrWriting',
    field: 'avatarUpload',
    extra: undefined } ]
[ { size: 251156,
    type: 'image/jpeg',
    filename: 'Cancelled-1.jpg',
    status: 'cancelled',
    field: 'avatarUpload',
    extra: undefined } ]
[ { size: 2337904,
    type: 'image/jpeg',
    filename: 'Success-2.jpg',
    status: 'bufferingOrWriting',
    field: 'avatarUpload',
    extra: undefined } ]
[ { size: 329172,
    type: 'image/jpeg',
    filename: 'Cancelled-2.jpg',
    status: 'cancelled',
    field: 'avatarUpload',
    extra: undefined } ]

It's also probably worth saying that if I don't comment out stream.pause() on line 33 of node_modules/skipper/node_modules/connect/node_modules/raw-data/index.js that I get this error.

Error: Cannot switch to old mode now.
    at emitDataEvents (_stream_readable.js:730:11)
    at IncomingMessage.Readable.pause (_stream_readable.js:721:3)
    at module.exports (/Users/Nate/Desktop/Projects/Active/project/node_modules/skipper/node_modules/connect/node_modules/raw-body/index.js:33:14)
    at json (/Users/Nate/Desktop/Projects/Active/project/node_modules/skipper/node_modules/connect/lib/middleware/json.js:51:5)
    at _parseHTTPBody (/Users/Nate/Desktop/Projects/Active/project/node_modules/skipper/index.js:71:5)
    at Object.retryBodyParser [as handle] (/Users/Nate/Desktop/Projects/Active/project/node_modules/sails/lib/express/bodyParserRetry.js:36:36)
    at next (/Users/Nate/Desktop/Projects/Active/project/node_modules/sails/node_modules/express/node_modules/connect/lib/proto.js:190:15)
    at next (/Users/Nate/Desktop/Projects/Active/project/node_modules/sails/node_modules/express/node_modules/connect/lib/proto.js:192:9)
    at Parser.next (/Users/Nate/Desktop/Projects/project/node_modules/skipper/index.js:97:20)
    at passControlToApp (/Users/Nate/Desktop/Projects/Active/project/node_modules/skipper/lib/Parser/parse.js:172:11)
    at null._onTimeout (/Users/Nate/Desktop/Projects/Active/project/node_modules/skipper/node_modules/async/lib/async.js:668:17)
    at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)

This only happens on the files that would otherwise be successful. If I upload one that'll be blank, then it works, but still writes an empty file.

I'm using Skipper 0.1.6 with Sails 0.9.8. But I've tried with Sails 0.9.16, with the same results

[tegandbiscuits]

I've noticed that both the pictures that work, have very descriptive metadata. The ones that make empty files, and one that hangs have pretty much no metadata. And the other one that hangs has some metadata. I could post the metadata (and/or the images) if it could be helpful.

[mikermcneil]

Definitely , that would be great

Mike's phone

On May 19, 2014, at 10:02, Nate Rauh notifications@github.com wrote:

I've noticed that both the pictures that work, have very descriptive metadata. The ones that make empty files, and one that hangs have pretty much no metadata. And the other one that hangs has some metadata. I could post the metadata (and/or the images) if it could be helpful.

— Reply to this email directly or view it on GitHub.

[tegandbiscuits]

Ok, here's the metadata from the images

Working images

---- File ----
File Name   Success-1.jpg
Directory   .
File Size   1958 kB
File Modification Date/Time 2014:01:04 19:27:54-06:00
File Access Date/Time   2014:05:19 17:09:50-05:00
File Inode Change Date/Time 2014:05:19 14:01:59-05:00
File Permissions    rw-r--r--
File Type   JPEG
MIME Type   image/jpeg
Exif Byte Order Little-endian (Intel, II)
Image Width 2272
Image Height    1704
Encoding Process    Baseline DCT, Huffman coding
Bits Per Sample 8
Color Components    3
Y Cb Cr Sub Sampling    YCbCr4:4:4 (1 1)
---- JFIF ----
JFIF Version    1.02
---- EXIF ----
Make    Canon
Camera Model Name   Canon PowerShot A520
Orientation Horizontal (normal)
X Resolution    180
Y Resolution    180
Resolution Unit inches
Y Cb Cr Positioning Centered
Software    Adobe Photoshop CS Windows
Modify Date 2007:09:03 18:27:53
Exposure Time   1/500
F Number    5.5
Exif Version    0220
Date/Time Original  2007:08:02 13:43:50
Create Date 2007:08:02 13:43:50
Components Configuration    Y, Cb, Cr, -
Compressed Bits Per Pixel   5
Shutter Speed Value 1/501
Aperture Value  5.5
Exposure Compensation   0
Max Aperture Value  5.5
Metering Mode   Multi-segment
Focal Length    23.2 mm
User Comment    
Flashpix Version    0100
Color Space sRGB
Exif Image Width    2272
Exif Image Height   1704
Interoperability Index  R98 - DCF basic file (sRGB)
Interoperability Version    0100
Related Image Width 2272
Related Image Height    1704
Focal Plane X Resolution    10142.85714
Focal Plane Y Resolution    10142.85714
Focal Plane Resolution Unit inches
Sensing Method  One-chip color area
File Source Digital Camera
Custom Rendered Normal
Exposure Mode   Auto
White Balance   Auto
Digital Zoom Ratio  1
Scene Capture Type  Landscape
Compression JPEG (old-style)
Thumbnail Offset    1260
Thumbnail Length    7075
---- Photoshop ----
IPTC Digest 00000000000000000000000000000000
Displayed Units X   inches
Displayed Units Y   inches
Global Angle    30
Global Altitude 30
Copyright Flag  False
Photoshop Thumbnail (Binary data 7075 bytes, use -b option to extract)
Photoshop Quality   11
Photoshop Format    Standard
Progressive Scans   3 Scans
---- XMP ----
XMP Toolkit XMP toolkit 3.0-28, framework 1.6
About   uuid:fc225bb7-59f6-11dc-b556-d0c050bf1a09
Date/Time Digitized 2007:08:02 13:43:50+10:00
Flash Fired False
Flash Return    No return detection
Flash Mode  Off
Flash Function  False
Flash Red Eye Mode  False
History 
Metadata Date   2007:09:03 18:27:53+10:00
Creator Tool    Adobe Photoshop CS Windows
Document ID adobe:docid:photoshop:fc225bb6-59f6-11dc-b556-d0c050bf1a09
Format  image/jpeg
---- ICC_Profile ----
Profile CMM Type    Lino
Profile Version 2.1.0
Profile Class   Display Device Profile
Color Space Data    RGB
Profile Connection Space    XYZ
Profile Date Time   1998:02:09 06:49:00
Profile File Signature  acsp
Primary Platform    Microsoft Corporation
CMM Flags   Not Embedded, Independent
Device Manufacturer IEC
Device Model    sRGB
Device Attributes   Reflective, Glossy, Positive, Color
Rendering Intent    Perceptual
Connection Space Illuminant 0.9642 1 0.82491
Profile Creator HP
Profile ID  0
Profile Copyright   Copyright (c) 1998 Hewlett-Packard Company
Profile Description sRGB IEC61966-2.1
Media White Point   0.95045 1 1.08905
Media Black Point   0 0 0
Red Matrix Column   0.43607 0.22249 0.01392
Green Matrix Column 0.38515 0.71687 0.09708
Blue Matrix Column  0.14307 0.06061 0.7141
Device Mfg Desc IEC http://www.iec.ch
Device Model Desc   IEC 61966-2.1 Default RGB colour space - sRGB
Viewing Cond Desc   Reference Viewing Condition in IEC61966-2.1
Viewing Cond Illuminant 19.6445 20.3718 16.8089
Viewing Cond Surround   3.92889 4.07439 3.36179
Viewing Cond Illuminant Type    D50
Luminance   76.03647 80 87.12462
Measurement Observer    CIE 1931
Measurement Backing 0 0 0
Measurement Geometry    Unknown (0)
Measurement Flare   0.999%
Measurement Illuminant  D65
Technology  Cathode Ray Tube Display
Red Tone Reproduction Curve (Binary data 2060 bytes, use -b option to extract)
Green Tone Reproduction Curve   (Binary data 2060 bytes, use -b option to extract)
Blue Tone Reproduction Curve    (Binary data 2060 bytes, use -b option to extract)
---- APP14 ----
DCT Encode Version  100
APP14 Flags 0   [14]
APP14 Flags 1   (none)
Color Transform YCbCr
---- Composite ----
Aperture    5.5
Flash   Off, Did not fire
Image Size  2272x1704
Scale Factor To 35 mm Equivalent    6.1
Shutter Speed   1/500
Thumbnail Image (Binary data 7075 bytes, use -b option to extract)
Circle Of Confusion 0.005 mm
Field Of View   14.5 deg
Focal Length    23.2 mm (35 mm equivalent: 141.1 mm)
Hyperfocal Distance 19.79 m

---- File ----
File Name   Success-2.jpg
Directory   .
File Size   2.2 MB
File Modification Date/Time 2013:10:31 15:17:17-05:00
File Access Date/Time   2014:05:19 14:02:02-05:00
File Inode Change Date/Time 2014:05:19 14:01:59-05:00
File Permissions    rw-r--r--
File Type   JPEG
MIME Type   image/jpeg
Exif Byte Order Big-endian (Motorola, MM)
Image Width 3264
Image Height    2448
Encoding Process    Baseline DCT, Huffman coding
Bits Per Sample 8
Color Components    3
Y Cb Cr Sub Sampling    YCbCr4:2:0 (2 2)
---- EXIF ----
Make    Apple
Camera Model Name   iPhone 5
Orientation Horizontal (normal)
X Resolution    72
Y Resolution    72
Resolution Unit inches
Software    7.0
Modify Date 2013:10:31 15:17:17
Y Cb Cr Positioning Centered
Exposure Time   1/60
F Number    2.4
Exposure Program    Program AE
ISO 50
Exif Version    0221
Date/Time Original  2013:10:31 15:17:17
Create Date 2013:10:31 15:17:17
Components Configuration    Y, Cb, Cr, -
Shutter Speed Value 1/60
Aperture Value  2.4
Brightness Value    5.45879242
Metering Mode   Spot
Flash   No Flash
Focal Length    4.1 mm
Sub Sec Time Original   715
Sub Sec Time Digitized  715
Flashpix Version    0100
Color Space sRGB
Exif Image Width    3264
Exif Image Height   2448
Sensing Method  One-chip color area
Scene Type  Directly photographed
Custom Rendered Unknown (3)
Exposure Mode   Auto
White Balance   Auto
Focal Length In 35mm Format 33 mm
Scene Capture Type  Standard
Lens Info   4.12mm f/2.4
Lens Make   Apple
Lens Model  iPhone 5 back camera 4.12mm f/2.4
GPS Latitude Ref    North
GPS Longitude Ref   West
GPS Altitude Ref    Above Sea Level
GPS Time Stamp  20:17:17.4
GPS Img Direction Ref   Magnetic North
GPS Img Direction   86.11938383
Compression JPEG (old-style)
Thumbnail Offset    1252
Thumbnail Length    11289
---- MakerNotes ----
Run Time Scale  1000000000
Run Time Epoch  0
Run Time Value  208809595202458
Run Time Flags  Valid
HDR Image Type  HDR Image
---- Composite ----
Aperture    2.4
Image Size  3264x2448
Run Time Since Power Up 2 days 10:00:09
Scale Factor To 35 mm Equivalent    8.0
Shutter Speed   1/60
Create Date 2013:10:31 15:17:17.715
Date/Time Original  2013:10:31 15:17:17.715
Thumbnail Image (Binary data 11289 bytes, use -b option to extract)
Circle Of Confusion 0.004 mm
Field Of View   57.2 deg
Focal Length    4.1 mm (35 mm equivalent: 33.0 mm)
Hyperfocal Distance 1.89 m
Light Value 9.4

Creates empty file

---- File ----
File Name   Cancelled-1.jpg
Directory   .
File Size   245 kB
File Modification Date/Time 2012:02:13 15:13:07-06:00
File Access Date/Time   2014:05:19 14:02:01-05:00
File Inode Change Date/Time 2014:05:19 14:01:59-05:00
File Permissions    rw-r--r--
File Type   JPEG
MIME Type   image/jpeg
Image Width 950
Image Height    650
Encoding Process    Baseline DCT, Huffman coding
Bits Per Sample 8
Color Components    3
Y Cb Cr Sub Sampling    YCbCr4:4:4 (1 1)
---- JFIF ----
JFIF Version    1.01
Resolution Unit inches
X Resolution    72
Y Resolution    72
---- Composite ----
Image Size  950x650

---- File ----
File Name   Cancelled-2.jpg
Directory   .
File Size   321 kB
File Modification Date/Time 2013:12:10 03:28:13-06:00
File Access Date/Time   2014:05:19 14:02:02-05:00
File Inode Change Date/Time 2014:05:19 14:01:59-05:00
File Permissions    rw-r--r--
File Type   JPEG
MIME Type   image/jpeg
Image Width 960
Image Height    1280
Encoding Process    Baseline DCT, Huffman coding
Bits Per Sample 8
Color Components    3
Y Cb Cr Sub Sampling    YCbCr4:4:4 (1 1)
---- JFIF ----
JFIF Version    1.01
Resolution Unit inches
X Resolution    72
Y Resolution    72
---- Composite ----
Image Size  960x1280

Hangs

---- File ----
File Name   Hangs-1.jpg
Directory   .
File Size   64 kB
File Modification Date/Time 2011:12:02 20:01:45-06:00
File Access Date/Time   2014:05:19 14:01:58-05:00
File Inode Change Date/Time 2014:05:19 14:01:59-05:00
File Permissions    rw-r--r--
File Type   JPEG
MIME Type   image/jpeg
Image Width 553
Image Height    738
Encoding Process    Baseline DCT, Huffman coding
Bits Per Sample 8
Color Components    3
Y Cb Cr Sub Sampling    YCbCr4:2:0 (2 2)
---- JFIF ----
JFIF Version    1.01
Resolution Unit None
X Resolution    1
Y Resolution    1
---- Composite ----
Image Size  553x738

---- File ----
File Name   Hangs-2.jpg
Directory   .
File Size   89 kB
File Modification Date/Time 2013:12:04 04:27:05-06:00
File Access Date/Time   2014:05:19 14:02:02-05:00
File Inode Change Date/Time 2014:05:19 14:01:59-05:00
File Permissions    rw-r--r--
File Type   JPEG
MIME Type   image/jpeg
Exif Byte Order Big-endian (Motorola, MM)
Image Width 600
Image Height    446
Encoding Process    Baseline DCT, Huffman coding
Bits Per Sample 8
Color Components    3
Y Cb Cr Sub Sampling    YCbCr4:2:0 (2 2)
---- EXIF ----
Orientation Horizontal (normal)
X Resolution    72
Y Resolution    72
Resolution Unit inches
Y Cb Cr Positioning Centered
Exif Version    0221
Components Configuration    Y, Cb, Cr, -
Flashpix Version    0100
Color Space sRGB
Exif Image Width    600
Exif Image Height   446
Scene Capture Type  Standard
Compression JPEG (old-style)
Thumbnail Offset    298
Thumbnail Length    11895
---- Composite ----
Image Size  600x446
Thumbnail Image (Binary data 11895 bytes, use -b option to extract)

[fabdrol]

@NRauh @mikermcneil I also have this issue, but I know for a fact that the files should upload normally. I used busboy before (in a quick express test backend for an app) which handled the files normally. Busboy code below, for reference:

var express     = require('express');
var bodyParser  = require('body-parser');
var busboy      = require('connect-busboy');
var app         = express();
var path        = require('path');
var util        = require('util');
var fs          = require('fs');

app.use(bodyParser.json()); // not related to upload
app.use(busboy());

app.post('/image', function handleUpload(req, res) {
    var fsstream, arr, ext;

    req.pipe(req.busboy);

    req.busboy.on('file', function (fieldname, file, filename) {
        arr     = filename.split('.');
        ext     = arr[arr.length - 1];
        fstream = fs.createWriteStream(__dirname + '/files/' + fieldname + '.' + ext);

        file.pipe(fstream);

        fstream.on('close', function () {
            console.log('POST /image', (fieldname + '.' + ext))
            res.send(fieldname + '.' + ext);
        });
    });
});

app.get('/file/:name', function retrieveFile(req, res) {
    var name = req.param('name');

    if(fs.existsSync(__dirname + '/files/' + name)) {
        return res.sendfile(__dirname + '/files/' + name);
    }

    res.send(404);
});

var server = app.listen(3000, function() {
    console.log('Listening on port %d', server.address().port);
});

[fabdrol]

@NRauh regarding the commenting out of line 33 in stream-utils/raw-body: I filed an issue in the raw-body repo, received this response:

you're probably using the stream somewhere else as well, which you shouldn't be doing (except for maybe pipes)

16 June 2014 12:38 - @jonathanong

https://github.com/stream-utils/raw-body/issues/23#issuecomment-46163639

[dougwilson]

The issue is from the "re-run of the JSON bodyparser" logic in skipper. The MultipartBodyParser did req.pipe but it did not call unpipe.

[dougwilson]

Can anyone here chime in on what version of multiparty is installed under their skipper install?

[dougwilson]

Basically in the current multiparty, the call at https://github.com/balderdashy/skipper/blob/master/lib/Parser/parse.js#L182 is irreversible, so when skipper's "re-run" logic then tries to parse after this, raw-body explodes.

[dougwilson]

Likely the easiest solution for you guys to do is do not try and parse non-multipart requests as mltipart by adding

if (!req.is('multipart/form-data')) return next();

on this line: https://github.com/balderdashy/skipper/blob/v0.1.8/lib/multipart.js#L19

[fabdrol]

@dougwilson shouldn't that be if ( ! req.is('multipart/form-data')) return next();, since that function is used to parse multipart forms.. returning next() would stop the rest of the method from running. Right?

Btw, the version of multiparty is ~3.2.2.

[dougwilson]

@fabdrol I'm not sure how the if statement you posted if different from mine except for the whitespace...?

Btw, the version of multiparty is ~3.2.2.

That's not a version; I know what this module requires, but that means if you be all kinds of versions. I want to know which one you actually have installed, since this module allows for a range of versions.

[fabdrol]

@dougwilson oops, sorry I didn't read yours properly. Sorry I posted that.

The version installed on my machine is 3.2.8.

[dougwilson]

Thanks. I can make a tweak to multiparty that will get this module to start functioning again, but the way this module does the re-run logic still needs to be changed to prevent stuff like this in the future.

[fabdrol]

@dougwilson I set up a fork of skipper at startingpoint/skipper (branch: raw_body_fix) which includes your line of code. I'll test it asap (tomorrow at the latest) in a real project.

[dougwilson]

Though the branch is a bit mis-named, because it's not a fix to raw-body, rather a fix to an issue in skipper that raw-body exposed.

[fabdrol]

well, yeah.. I guess. But it's just a name, right

[mikermcneil]

Thanks. I can make a tweak to multiparty that will get this module to start functioning again, but the way this module does the re-run logic still needs to be changed to prevent stuff like this in the future.

@dougwilson thanks! I'll tackle that asap

@NRauh @fabdrol thanks for helping figure this out guys

[fabdrol]

@mikermcneil np let me know if I can help in one way or another

[fabdrol]

P.s.: this thread was originally started regarding an issue with Cancelled uploads, I reckon that is a different problem alltogether. Any ideas about that?

[dougwilson]

@fabdrol did you have a chance to test it on your branch?

[fabdrol]

@dougwilson actually working on that as we speak :)

[fabdrol]

@dougwilson did some testing, and I'm still getting the error when uploading something using multipart. No errors when sending "normal" requests.

Error: Cannot switch to old mode now.
    at emitDataEvents (_stream_readable.js:730:11)
    at IncomingMessage.Readable.pause (_stream_readable.js:721:3)
    at module.exports (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/node_modules/connect/node_modules/raw-body/index.js:33:14)
    at json (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/node_modules/connect/lib/middleware/json.js:51:5)
    at _parseHTTPBody (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/index.js:71:5)
    at Object.retryBodyParser [as handle] (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/lib/express/bodyParserRetry.js:36:36)
    at next (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/node_modules/express/node_modules/connect/lib/proto.js:190:15)
    at next (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/node_modules/express/node_modules/connect/lib/proto.js:192:9)
    at Parser.next (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/index.js:97:20)
    at passControlToApp (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/lib/Parser/parse.js:172:11)
    at null._onTimeout (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/async/lib/async.js:668:17)
    at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)
debug: Lowering sails...

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: stream ended unexpectedly
    at Form.<anonymous> (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/node_modules/multiparty/index.js:664:24)
    at Form.EventEmitter.emit (events.js:117:20)
    at finishMaybe (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/node_modules/multiparty/node_modules/readable-stream/lib/_stream_writable.js:455:14)
    at afterWrite (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/node_modules/multiparty/node_modules/readable-stream/lib/_stream_writable.js:335:3)
    at onwrite (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/node_modules/multiparty/node_modules/readable-stream/lib/_stream_writable.js:325:7)
    at WritableState.onwrite (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/node_modules/multiparty/node_modules/readable-stream/lib/_stream_writable.js:110:5)
    at process._tickDomainCallback (node.js:459:13)

[dougwilson]

@fabdrol are you sure you are actually using the code from that branch you made? The line numbers in the stack trace aren't quite matching up with the code in your fork.

[fabdrol]

Which line numbers don't match?

The screen shot with the code change:

[dougwilson]

Hm, IDK. You'll have to debug this and figure out the solution, as I don't have any time to. I can tell you the error that comes from raw-body is because the req was piped somewhere (probably the multipart parser) and was never unpiped before skipper's "retry logic" then gave the req to raw-body, which is an error.

If you want to given me the output for console.trace() added to the line https://github.com/startingpoint/skipper/blob/raw_body_fix/lib/Parser/parse.js#L18 I can see how the parser is getting invoked by skipper.

[fabdrol]

Trace
    at Parser.parseRequest [as parse] (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/lib/Parser/parse.js:18:10)
    at new Parser (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/lib/Parser/index.js:41:7)
    at _parseMultipartHTTPRequest (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/lib/multipart.js:31:34)
    at /Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/index.js:75:9
    at urlencoded (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/node_modules/connect/lib/middleware/urlencoded.js:43:72)
    at /Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/index.js:73:7
    at json (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/node_modules/connect/lib/middleware/json.js:45:55)
    at _parseHTTPBody (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/skipper/index.js:71:5)
    at Object.handle (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/lib/express/index.js:118:12)
    at next (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/node_modules/express/node_modules/connect/lib/proto.js:190:15)
    at Object.session [as handle] (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/node_modules/express/node_modules/connect/lib/middleware/session.js:301:7)
    at next (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/node_modules/express/node_modules/connect/lib/proto.js:190:15)
    at Object.cookieParser [as handle] (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/node_modules/express/node_modules/connect/lib/middleware/cookieParser.js:60:5)
    at next (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/node_modules/express/node_modules/connect/lib/proto.js:190:15)
    at Object.expressInit [as handle] (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/node_modules/express/lib/middleware.js:30:5)
    at next (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/node_modules/express/node_modules/connect/lib/proto.js:190:15)
    at Object.query [as handle] (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/node_modules/express/node_modules/connect/lib/middleware/query.js:44:5)
    at next (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/node_modules/express/node_modules/connect/lib/proto.js:190:15)
    at Function.app.handle (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/node_modules/express/node_modules/connect/lib/proto.js:198:3)
    at Server.app (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/node_modules/express/node_modules/connect/lib/connect.js:65:37)
    at Manager.handleRequest (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/node_modules/socket.io/lib/manager.js:565:28)
    at Server.<anonymous> (/Volumes/Storage/Fabian/Dropbox/Sites/Klanten/Novanet/AppServer/node_modules/sails/node_modules/socket.io/lib/manager.js:119:10)
    at Server.EventEmitter.emit (events.js:98:17)
    at HTTPParser.parser.onIncoming (http.js:2108:12)
    at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:121:23)
    at Socket.socket.ondata (http.js:1966:22)
    at TCP.onread (net.js:525:27)

[dougwilson]

Awesome, thanks. And what are you actually posting when this occurs? Is it actually multipart/form-data?

[fabdrol]

yeah, it's an image upload from a mobile app (Titanium, running in simulator)

[dougwilson]

OK, then my change won't change anything there. I was assuming you guys were not submitting multipart. If you are submitting multipart data, then the "retry logic" should never even be invoked, so it seems like that is the issue, which is completely different than what I was thinking here. I don't have any more information, unfortunately, since I don't actually use this library :(

[fabdrol]

@dougwilson no problem, thanks for your input! We'll get there, eventually :)

[mikermcneil]

@fabdrol any updates on this from your end? I can't replicate any issues using the latest version of skipper, so it could be a version issue (I might also be missing something).

@dougwilson thanks for your help!

[fabdrol]

@mikermcneil not yet, same issues as both @NRauh and I had originally.. I can't upgrade to 10 anytime soon, in this project. @NRauh, did you ever solve this/found out what exactly causes the issue?

Something I noticed: it fails every time when I upload an image taken with my iPhone's camera and then saved to the camera roll. It does not, however, fail when I take a picture and upload it directly. Also, if I save a picture to the camera roll (on my iphone) from the internet, results are mixed.

[tegandbiscuits]

@fabdrol @mikermcneil I was never able to figure out what's causing the problem, or an actual solution. I was able to upgrade my project to 10, and it worked.

I wish I had a better solution.

[ghost]

I'm having the same issue. Files larger than ~70KB are cancelled. Details here: http://stackoverflow.com/questions/24683600/file-upload-is-not-working-on-sails-js

[fabdrol]

Okay, this is becoming a real issue..

@mikermcneil, could you give me some pointers to where the the bodyParser resides within the sails stack/how I would go about hooking in my own bodyParser? (I'm not actually planning to write another bodyParser, i just need some more bg info in order to be able to contribute to Skipper in a useful way)

[Arsca]

@arctouch-ecilteodoro, I confirm your observation. I have the very same situation here. Trying to pipe to Mongo GrifFS and only files <70kb are received OK. Using req.file(req.body.ficheroNombre).pipe(receiver).on(..)

[ghost]

@Arsca, I use Postman to test my endpoints and I always get this error. Then a team mate built an app to upload the photo and it works fine. So iOS client, Android client and web client are working fine. Postman client does not work. What client are you using?

[Arsca]

@arctouch-ecilteodoro, I'm doing it directly from the webclient GUI.

[Arsca]

@arctouch-ecilteodoro, I figured it out and opened an issue: https://github.com/balderdashy/skipper/issues/24

[mikermcneil]

@Arsca, I use Postman to test my endpoints and I always get this error. Then a team mate built an app to upload the photo and it works fine. So iOS client, Android client and web client are working fine. Postman client does not work. What client are you using

hmm- could be headers? I use postman to test as well and haven't had trouble w/ this. You do need to set the multipart content-type header though

[mikermcneil]

But I've tried with Sails 0.9.16, with the same results

Would you mind trying with Sails v0.10.0-rc11?

sudo npm install sails@beta -g

[ecteodoro]

This is the request preview from Postman:

POST /customer/1/photo HTTP/1.1
Host: localhost:1337
x-access-token: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VybmFtZSI6ImVjaWwudGVvZG
Cache-Control: no-cache
Postman-Token: 9e7b2e30-fc60-4d40-4b9e-8e68e27da411
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryp7MA4YWxkTrZu0gW

----WebKitFormBoundaryE19zNvXGzXaLvS5C
Content-Disposition: form-data; name="photo"; filename="03957c2.jpg"
Content-Type: image/jpeg

----WebKitFormBoundaryE19zNvXGzXaLvS5C

[mikermcneil]

@ectedoro Hm, so are you getting etimeout or emaxbuffer?

[mikermcneil]

I've noticed that both the pictures that work, have very descriptive metadata. The ones that make empty files, and one that hangs have pretty much no metadata.

@NRauh so that would mean that, in those cases, skipper gets a longer window to "hear about" incoming file params since there it has to buffer more text param data first

[ecteodoro]

@mikermcneil None. It's working now. I did not upgrade Sails, so I'm not sure what happened.

[uncletammy]

@NRauh, @ecteodoro

I'm going to close this out for now. Feel free to continue discussion or submit a PR to Roadmap.md with a feature request.

[ArVan]

Don't know if it's the same issue, but I do experience 0 byte files being written very often. In fact some time the same file uploads normally, and then sometime later it doesn't. The file is not being written especially when uploading multiple files. Then the 1st one is saved, and the 2nd one is not.

Here is the request payload which failed:

------WebKitFormBoundary4lserNai1ZrvGIIS
Content-Disposition: form-data; name="_csrf"

mltt6zttqawX379V+rxDOQCKX+CFrT9gfH5Cs=
------WebKitFormBoundary4lserNai1ZrvGIIS
Content-Disposition: form-data; name="profile_photo"; filename="agentsmith.jpg"
Content-Type: image/jpeg

------WebKitFormBoundary4lserNai1ZrvGIIS
Content-Disposition: form-data; name="company_logo"; filename="matrix.jpg"
Content-Type: image/jpeg

------WebKitFormBoundary4lserNai1ZrvGIIS--

[uncletammy]

@ArVan, can you post sails version information along with the controller code so I can try to replicate it? Thanks.

[ArVan]

I'm on Sails v0.10.4 containing Skipper v0.5.3 And here's the controller code:

create: function (req, res) {
        var userId = req.session.theUser ? req.session.theUser.id : req.param('user_id');

        if (!userId) {
            return res.status(400).json({title: HTTPStatusText[res.statusCode], code: ErrorCodes.WRONG_PARAMS, detail: ErrorMessages.WRONG_PARAMS});
        }

        var fileParamNames = req.param('_fileParamNames');

        if(!fileParamNames) {
            return res.status(201).json({title: HTTPStatusText[res.statusCode], data: {filesJSON: filesJSON}});
        }

         var filesJSON = [];

        async.eachSeries(fileParamNames, function(file, cb) {

            req.file(file).upload(function (err, files) {
                if (err) {
                    return cb(err);
                }

                sails.controllers['site/photo']._save(files, userId, req, filesJSON, 0, function (err, newFilesJSON) {
                    if (err) {
                        return cb(err);
                    }

                    filesJSON = newFilesJSON;
                    return cb();
                }, file);
            });

        }, function doneUploading(err) {

            if (err) {
                return res.serverError(err);
            }

            return res.status(201).json({title: HTTPStatusText[res.statusCode], data: {filesJSON: filesJSON}});

        });
    }

in the _save function I just read the file and apply imagemagick on it. The file is already empty there.

[ArVan]

I have debugged and found the following in skipper-disk module. build-disk-receiver-stream.js Line 74:

logged __newFile, and saw that for uploaded files:

normal upload

{ _readableState: 
   { highWaterMark: 16384,
     buffer: 
      [ <Buffer ff d8 ff e0 00 10 4a 46 49 46 00 01 01 01 01 2c 01 2c 00 00 ff db 00 43 00 08 06 06 07 06 05 08 07 07 07 09 09 08 0a 0c 14 0d 0c 0b 0b 0c 19 12 13 0f 14 ...>,
        <Buffer 0d>,
        <Buffer 0d>,
        <Buffer 18 32 21 1c 21 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 ...> ],
     length: 12302,
     pipes: null,
     pipesCount: 0,
     flowing: null,
     ended: true,
     endEmitted: false,
     reading: false,
     sync: false,
     needReadable: false,
     emittedReadable: true,
     readableListening: false,
     objectMode: false,
     defaultEncoding: 'utf8',
     ranOut: false,
     awaitDrain: 0,
     readingMore: false,
     decoder: null,
     encoding: null },
  readable: true,
  domain: null,
  _events: 
   { end: { [Function: g] listener: [Function: onend] },
     drain: [Function],
     error: { [Function: g] listener: [Function] } },
  _maxListeners: 10,
  _writableState: 
   { highWaterMark: 16384,
     objectMode: false,
     needDrain: false,
     ending: true,
     ended: true,
     finished: true,
     decodeStrings: true,
     defaultEncoding: 'utf8',
     length: 0,
     writing: false,
     corked: 0,
     sync: false,
     bufferProcessing: false,
     onwrite: [Function],
     writecb: null,
     writelen: 0,
     buffer: [],
     pendingcb: 0,
     prefinished: true,
     errorEmitted: false },
  writable: true,
  allowHalfOpen: true,
  _transformState: 
   { afterTransform: [Function],
     needTransform: false,
     transforming: false,
     writecb: null,
     writechunk: null,
     writeencoding: 'buffer' },
  headers: 
   { 'content-disposition': 'form-data; name="profile_photo"; filename="agentsmith.jpg"',
     'content-type': 'image/jpeg' },
  name: 'profile_photo',
  filename: 'agentsmith.jpg',
  byteOffset: 281,
  byteCount: 1240221,
  field: 'profile_photo',
  fd: '4321e8ec-94ef-48ec-932d-f486638a8032.jpg' }

0 byte upload

{ _readableState: 
   { highWaterMark: 16384,
     buffer: [],
     length: 0,
     pipes: null,
     pipesCount: 0,
     flowing: false,
     ended: true,
     endEmitted: true,
     reading: false,
     sync: false,
     needReadable: false,
     emittedReadable: false,
     readableListening: false,
     objectMode: false,
     defaultEncoding: 'utf8',
     ranOut: false,
     awaitDrain: 0,
     readingMore: false,
     decoder: null,
     encoding: null,
     resumeScheduled: false },
  readable: false,
  domain: null,
  _events: { drain: [Function] },
  _maxListeners: 10,
  _writableState: 
   { highWaterMark: 16384,
     objectMode: false,
     needDrain: false,
     ending: true,
     ended: true,
     finished: true,
     decodeStrings: true,
     defaultEncoding: 'utf8',
     length: 0,
     writing: false,
     corked: 0,
     sync: false,
     bufferProcessing: false,
     onwrite: [Function],
     writecb: null,
     writelen: 0,
     buffer: [],
     pendingcb: 0,
     prefinished: true,
     errorEmitted: false },
  writable: true,
  allowHalfOpen: true,
  _transformState: 
   { afterTransform: [Function],
     needTransform: false,
     transforming: false,
     writecb: null,
     writechunk: null,
     writeencoding: 'buffer' },
  headers: 
   { 'content-disposition': 'form-data; name="company_logo"; filename="matrix.jpg"',
     'content-type': 'image/jpeg' },
  name: 'company_logo',
  filename: 'matrix.jpg',
  byteOffset: 12731,
  byteCount: 1227771,
  field: 'company_logo',
  fd: 'c791bc27-e503-4c16-af34-a2ffc3c95e64.jpg' }

See, the buffer is empty for the failed file. How come this happen?