dougwilson
commented
3 years ago Unfortunately express itself is not involved in the transfer of bytes; express is a routing layer on top of the Node.js http server. The various modules you are using read the bytes directly from the Node.js http server; express is not actually involved, unfortunately. If you are not getting the entire file, it is either something wrong in Node.js http server or in one of the intermediate proxies.
If you can provide an example app and the exact steps to reproduce the issue with the app, I can help track down what is happening, though, even if that is ultimately to report a bug to Node.js. Without a reproduction case provided, the best I can say is that I run a site that does something very similar to what you described (uses multiparty for the parsing) and have never had such an issue.
nickgermaine
The issue: I have an app in production where the front end is React, and the api is express. One of the app pages is a video upload page, that sends the file to the api (both are served through nginx). The video file (as written to the /tmp dir) is then loaded into fluent ffmpeg and converted, uploaded to storage server, etc.
I began getting reports that the upload function was not working. Digging into this, I took note of the file size transmitted from the react app, and the size of the file received in express. The react app always shows that sent bytes === total bytes.
The express api shows (sometimes -- specifically in instances where the function later fails in the conversion step), that not all bytes are always received. It happens about half the time. Sometimes the entire file is received.
Initially I was using multiparty, but in my troubleshooting I've replaced that with busboy, then formidable, and the same thing is happening with all of those. I'm not sure if it is, but it seems like an issue with express. Currenty using express 4.16.4
I'm experiencing the issue on multiple servers. The production server, AND a beta server, AND my local development pc.
OS: Ubuntu 20.04 (both servers) Windows 10 (development pc) macOS 12 (other developmetn pc) Running express with: pm2 (servers), npm start (pc) web server: nginx (servers), none (pc)
The specific issue that occurs is during the ffmpeg conversion, it errors out with "invalid data found while processing this file" because some important flags are usually stored at the end of video files, and that part of the file isn't available. Videos range in size from 15mb to 300mb and doesn't seem to have impact on the failure rate of this specific route.
I know there's a lot of variables here, but I'm hoping someone can help me with this. I have tried EVERYTHING (I can think of), and nothing is fixing it.
Here is my package.json file if that is helpful.. Idunno.
{ "name": "f21-api", "version": "0.0.0", "private": true, "scripts": { "start": "node ./bin/www" }, "dependencies": { "@ffmpeg-installer/ffmpeg": "^1.1.0", "@socket.io/sticky": "^1.0.1", "aws-cloudfront-sign": "^2.2.0", "aws-sdk": "^2.951.0", "bcrypt": "^5.0.1", "busboy": "^0.3.1", "cluster": "^0.7.7", "cookie-parser": "~1.4.4", "cors": "^2.8.5", "dayjs": "^1.10.6", "debug": "~2.6.9", "express": "~4.16.1", "express-busboy": "^8.0.0", "express-fileupload": "^1.2.1", "express-formidable": "^1.2.0", "express-ws-routes": "^1.1.0", "farmhash": "^3.2.1", "ffmpeg-extract-frames": "^2.0.2", "ffprobe": "^1.1.2", "ffprobe-static": "^3.0.0", "fluent-ffmpeg": "^2.1.2", "formidable": "git+https://github.com/node-formidable/formidable.git", "get-video-dimensions": "^1.0.0", "hls-ffmpeg": "0.0.9", "http-errors": "~1.6.3", "jsonwebtoken": "^8.5.1", "lodash": "^4.17.21", "mandrill-api": "^1.0.45", "mongoose": "^5.13.3", "morgan": "~1.9.1", "multiparty": "^4.2.2", "nodemailer": "^6.6.3", "pdfkit": "^0.11.0", "redis": "^3.1.2", "sharp": "^0.25.4", "shelljs": "^0.8.4", "socket.io": "^3.1.2", "socket.io-redis": "^6.1.1", "stripe": "^8.164.0", "ws": "^7.5.3", "xlsx-to-json": "^0.3.0", "xml2js": "^0.4.23" } }