Parse.File Terminates when Encoding/Decoding Large Media File(s) with URI Constructor

HackShitUp commented 3 months ago

New Issue Checklist

[X] I am not disclosing a vulnerability.
[X] I am not just asking a question.
[X] I have searched through existing issues.
[X] I can reproduce the issue with the latest versions of Parse Server and the Parse JS SDK.

Issue Description

Note: This issue has already been tested with the following:

Configuring ParseServer options maxUploadSize

Defining directives for an NGINX proxy

For applications handling large media data, performing either chunked uploads or directly writing to an independent 3rd party storage provider via the client is advised. For the latter, assuming one's correctly retrieved the remote url of the uploaded media file, documentation implies it's possible to construct a new Parse.File object with the following:

new Parse.File(`fileName`, {uri:  "https://urlToLargeMediaFile"})

which can then be set on a Parse.Object. However, Parse.File seems to automatically attempt to encode the data and terminates ParseServer with RangeError: Invalid string length.

Steps to reproduce

Upload large video file to some external storage provider
Construct new Parse.File(fileName, {uri:remoteURLToLargeMediaFile})
Attempt to save Parse.File and set to Parse.Object

Actual Outcome

ParseServer terminates with the following error:

RangeError: Invalid string length
    at IncomingMessage.<anonymous> (/Users/John/Project/test/node_modules/parse-server/node_modules/parse/lib/node/ParseFile.js:468:45)
    at IncomingMessage.emit (node:events:513:28)
    at IncomingMessage.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:293:11)
    at Readable.push (node:internal/streams/readable:234:10)
    at HTTPParser.parserOnBody (node:_http_common:131:24)
    at TLSSocket.socketOnData (node:_http_client:542:22)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Readable.push (node:internal/streams/readable:234:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)

Expected Outcome

Ideally, we should extend Parse.File to simply be initialized without attempting to encode/decode the data specified by the uri. For now, I've managed to define the file property of a Parse.Object with the following:

object.set(`file`, {
    "__type": "File",
    "name": "demo.mp4",
    "url": "https://urlToRemoteMediaFile/tag.mp4"
})

Not sure if there's some further configuration one can do within the files adapter, but either way, I thought this was an issue to raise when handling memory allocation with large media data...

Server

Parse Server version: 7.0.0
Operating system: MacOS
Local or remote host (AWS, Azure, Google Cloud, Heroku, Digital Ocean, etc): LocalHost/Google Cloud

Database

System (MongoDB or Postgres): Postgres
Database version: 16.3
Local or remote host (MongoDB Atlas, mLab, AWS, Azure, Google Cloud, etc): Postgres

Client

Parse JS SDK version: 5.1.0

parse-github-assistant[bot] commented 3 months ago

Thanks for opening this issue!

🚀 You can help us to fix this issue faster by opening a pull request with a failing test. See our Contribution Guide for how to make a pull request, or read our New Contributor's Guide if this is your first time contributing.

mtrezza commented 3 months ago

Ideally, we should extend Parse.File to simply be initialized without attempting to encode/decode the data specified by the uri.

Attempt to save Parse.File and set to Parse.Object

Does this error occur when you only init Parse.File, or when you actually save it? If you save the Parse.File, then it has to be downloaded from the external storage to store it in the Parse Server's storage provider, right?

I classified this as a bug, but I'm not sure it is one at this point. If the file is ~very large, then this may be a resource limitation and there may be other ways (multi-part, streaming) to handle this.

HackShitUp commented 3 months ago

Ideally, we should extend Parse.File to simply be initialized without attempting to encode/decode the data specified by the uri.

Attempt to save Parse.File and set to Parse.Object

Does this error occur when you only init Parse.File, or when you actually save it? If you save the Parse.File, then it has to be downloaded from the external storage to store it in the Parse Server's storage provider, right?

I classified this as a bug, but I'm not sure it is one at this point. If the file is ~very large, then this may be a resource limitation and there may be other ways (multi-part, streaming) to handle this.

Thanks for the quick reply.

This only occurs when you initialize a Parse.File then save Parse.Object with it. In retrospect, this should be flagged as a feature request given that one might still want to set a Parse.File object with a remote URL with the already written media content.

mtrezza commented 3 months ago

You want to save a Parse File with an external URL, without actually storing the file data in the Parse Server's storage? In other words, the file would point to an external storage that is not managed by Parse Server via a storage adapter, right? I don't think this is currently supported, because it would pose several questions, e.g. what happens if delete is called on the file?

I'm not sure this should even be supported. Parse File is an object for storage managed by Parse Server. If you don't want Parse Server to manage the storage, then just store the URL as a string in a normal Parse Object.

HackShitUp commented 3 months ago

You want to save a Parse File with an external URL, without actually storing the file data in the Parse Server's storage? In other words, the file would point to an external storage that is not managed by Parse Server via a storage adapter, right? I don't think this is currently supported, because it would pose several questions, e.g. what happens if delete is called on the file?

I'm not sure this should even be supported. Parse File is an object for storage managed by Parse Server. If you don't want Parse Server to manage the storage, then just store the URL as a string in a normal Parse Object.

I already store the URL in an afterSave trigger request by retrieving the url() property of a Parse.File — it's just that one might still want to use Parse.File for handling smaller media files and maintain consistent DB schema.

What I'm alluding to is extending the Parse.File object to be saved with a URL that's managed by the same files/storage provider given that Parse.File can't write large media files on the backend.

For example, if you've got a mobile application that's handling large/small media files, the smaller ones could be sent to a cloud code function; the larger ones are written to GC or S3 then sent back to a cloud code function to save the URL as an attribute.

mtrezza commented 3 months ago

To summarize: you create a Parse.File (for example from JSON) that points to a file that already exists in the Parse Server managed storage, and then save the Parse.File as property of a Parse.Object. And you observe that Parse Server downloads the whole file when saving the Parse.Object, but you want to avoid that. Is that it?

I agree that Parse Server downloading the whole file is quite an expensive operation and somewhat unexpected. I assume it's to determine the MIME type. Did you take a look at the code to see why it's downloading?

parse-community / Parse-SDK-JS