File upload sometimes hangs - multer, busboy, or something else?

[filipabramowicz]

HI!

I encountered some strange problem while using multer - sometimes it hangs while uploading the file.

When I am testing this behavior on my Ubuntu it succeeds every time, but when I switch to development server it hangs quite often.

I compared the behavior on both environments and added some basic printouts for troubleshooting:

app.use(multer({
  dest: './uploads/',
  rename: function (fieldname, filename) {
    return filename.replace(/\W+/g, '-').toLowerCase();
  },
  onFileUploadStart: function (file) {
    process.stderr.write('Uploading file..........');
  },
  onFileUploadComplete: function (file) {
    process.stderr.write('done\n');
  },

...

}));

On development server Uploading file......... is printed in the console but done doesn't show up - which suggests it can be multer or busboy issue.

Full code in which I am using multer can be found here: https://github.com/aplikacjespoleczne/WizualizacjaBudzetu

Also for troubleshooting I've added the simmilar logging to the multer code (in index.js) :

        fileStream.on('data', function(data) {
          console.log('Next part received!');        // ADDED
          if (data) { file.size += data.length; }
          // trigger "file data" event
          if (options.onFileUploadData) { options.onFileUploadData(file, data); }
        });

And the difference between Ubuntu and development server is the last chunk of data. In both environments Next part received! shows up but on development server done doesn't - which may suggest that the 'finish' event is not emitted.

Can it be multer or busboy? Can you give me some advice where to look further? What else could I trace?

Thanks!

[zhaoxingguang14]

I'm solved my question above, and just got clearly problem is about proxy reverse timeout. Even right know i'm still faced a prob with larger file but i think it's not on this Multer Issues, thanks :+1:

[gurumelo]

I note that when it fails. Does not get any request to express.

i have this problem. no nginx. pure node. randomly up or not. When i exec at first time, fails. Sometimes.

var confi               = require("./confi.json");
var express             = require("express");
var bodyParser          = require("body-parser");
var valida              = require("validator");
var app                 = express();
var router              = express.Router();
var mongo               = require("./model/mongo");
var shortid             = require("shortid");

var multer              = require('multer');
var storage             = multer.diskStorage({
  destination: function (req, file, callback) {
    callback(null, 'public/fotos');
  },
  filename: function (req, file, callback) {
    callback(null, shortid.generate() +'-'+ Date.now() +'.jpg');
  }
});
var upload              = multer({
    storage : storage,
    limits: { fileSize: 72351744 },
    fileFilter: function (req, file, cb) {

                                if (file.mimetype !== 'image/jpeg' || file.originalname !== 'imagen.jpg') {
                                req.fileValidationError = 'error';
                                return cb(null, false, new Error('error'));
                        }
                        cb(null, true);

    }

}).single('file');

router.post("/fotoolor", function (req, res) {
                        upload(req,res,function(err) {
                                if (err || req.fileValidationError) {
                                        res.json({"error": true});
                                        console.log('err');
                                }
                                res.json({"error": false});
                                console.log(req.file.path);
                                console.log(req.body);
                        });

});

I upload with cordova filetransfer.

                    function subida(r) {
                            console.log(r);
                    }

                    function fallosubida(error) {
                            console.log(error);
                    }

                    //console.log(rutaimagen);
                    var opciones = new FileUploadOptions();
                    opciones.fileKey = 'file';
                    opciones.fileName = 'imagen.jpg';
                    opciones.mimeType = 'image/jpeg';

                    opciones.params = losdatos;

                    var ft = new FileTransfer();
                    ft.upload(rutaimagen, encodeURI('http://url:3000/fotoolor'), subida, fallosubida, opciones);

I upload files between 50 - 100 KB

[makeitrein]

@a9urv may whatever deity you believe in grant you a boon for your bug fix

6 hours of debugging before switching formData order, fook me but it worked!

[a9urv]

@makeitrein - don't believe in any, but thanks!

[Gilthans]

If you're using Next.Js and this is happening to you, make sure you have the api bodyParser configured to false (see example here). Hope this saves someone else the time it took me...

[motdde]

Hello, @Gilthans,

I am curious to see what your handler looks like.

I am still having these issues even with bodyParser configured to false.

[Gilthans]

Something along the lines of this:

const upload = multer({
    storage: multer.diskStorage({
        // ...
    }),
});
export default nextConnect<NextApiRequest, NextApiResponse>({
    onNoMatch(req: NextApiRequest, res: NextApiResponse) {
        res.status(405).json({ error: `Method '${req.method}' Not Allowed` });
    },
    onError(err, req: NextApiRequest, res: NextApiResponse, next) {
        console.log('Failure');
        console.log(err);
        res.status(500).send(err.toString());
    },
})
    .use(AuthenticationMiddleware())
    .use(upload.single('file'))
    .post(async (req, res) => {
        // ...
    });
export const config = { api: { bodyParser: false } };

[motdde]

Ah, interesting.

Are you making use of the next connect package?

I have a global error handler and here is what mine looks like:

...

type response = {
  status: boolean
  message?: string
  data?: object
}

export default apiHandler({ post: handler })

async function handler(
  req: NextApiRequest & { [key: string]: any },
  res: NextApiResponse<response>
) {
  const currentUser = verifyUserToken(req, res)

  const upload = multer({
    storage: multer.memoryStorage(),
    limits: { files: 1, fileSize: 2000000 },
  })

  const multipleUpload = upload.fields([
    { name: 'profilePicture', maxCount: 1 },
  ])

  await runMiddleware(req, res, multipleUpload)

  console.log(req.files)

  return res
    .status(200)
    .json({ status: true, message: 'profile created successfully' })
}

export const config = {
  api: {
    bodyParser: false,
  },
}

function runMiddleware(
  req: NextApiRequest & { [key: string]: any },
  res: NextApiResponse,
  fn: (...args: any[]) => void
): Promise<any> {
  return new Promise((resolve, reject) => {
    fn(req, res, (result: any) => {
      if (result instanceof Error) {
        return reject(result)
      }

      return resolve(result)
    })
  })
}

Multer still hangs and this route is unable to process any other request if an error occurs.

[motdde]

@Gilthans Do you also experience this issue using next connect?