Parse.Files cannot be created if over 512MB

dalyaidan1 commented 3 weeks ago

New Issue Checklist

Report security issues confidentially.
Any contribution is under this license.
Before posting search existing issues.

Issue Description

Files larger than 512MB are not able to be uploaded to Parse Server as Parse.Files.

The files are converted to stings, which is more than Node can handle.

Steps to reproduce

Upload a file bigger than 512MB.

Actual Outcome

Error: Cannot create a string longer than 0x1fffffe8 characters

    at Object.slice (node:buffer:654:37)

    at Buffer.toString (node:buffer:824:14)

    at createHandler (/project/server/node_modules/parse-server/lib/Routers/FilesRouter.js:153:29)

    at Layer.handle [as handle_request] (/project/server/node_modules/express/lib/router/layer.js:95:5)

    at next (/project/server/node_modules/express/lib/router/route.js:144:13)

    at handleParseSession (/project/server/node_modules/parse-server/lib/middlewares.js:281:7)

    at Layer.handle [as handle_request] (/project/server/node_modules/express/lib/router/layer.js:95:5)

    at next (/project/server/node_modules/express/lib/router/route.js:144:13)

    at handleRateLimit (/project/server/node_modules/parse-server/lib/middlewares.js:275:3)

    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {

  code: 'ERR_STRING_TOO_LONG'

This appears to come from this line: https://github.com/parse-community/parse-server/blob/6.3.1/src/Routers/FilesRouter.js#L175

Expected Outcome

The Parse.File to be created.

Environment

Server

Parse Server version: 6.3.1
Operating system: Linux
Local or remote host (AWS, Azure, Google Cloud, Heroku, Digital Ocean, etc): Aptible

Database

System (MongoDB or Postgres): MongoDB
Database version: 4.0
Local or remote host (MongoDB Atlas, mLab, AWS, Azure, Google Cloud, etc): Aptible

Client

SDK (iOS, Android, JavaScript, PHP, Unity, etc): Android
SDK version: 4.2.0

Logs

parse-github-assistant[bot] commented 3 weeks ago

Thanks for opening this issue!

🚀 You can help us to fix this issue faster by opening a pull request with a failing test. See our Contribution Guide for how to make a pull request, or read our New Contributor's Guide if this is your first time contributing.

mtrezza commented 3 weeks ago

This is an expected limitation of the V8 engine. Large files should be uploaded in a different way, like streaming or multi-part. Not sure what the file adapter already supports. A "workaround" could be to break the base64 conversion up into parts instead of converting the whole string at once. But even then a var would hold the entire file in memory, which may again run into limitations. Maybe the conversion can be made unnecessary if req.body is already a buffer and we could simply const file = new Parse.File(filename, { buffer: fileBuffer }, contentType);.

parse-community / parse-server