search

joltup / rn-fetch-blob

A project committed to making file access and data transfer easier, efficient for React Native developers.
MIT License
2.85k stars 787 forks source link

Large multipart uploads (iOS) #487

Open cristianoccazinsp opened 5 years ago

cristianoccazinsp commented 5 years ago

rn-fetch-blob: 0.11.2

Hello all, this is mainly for iOS, but I have a feeling that Android has the same issue.

I'm wonder what would be the best way to do a multipart-form upload with large files. The idea is to upload key-value data, together with a file. However, iOS does not seem to allow large uploads with the current implementation when multi-part is used. More specifically, this line here (https://github.com/joltup/rn-fetch-blob/blob/master/ios/RNFetchBlobReqBuilder.m#L235) basically builds the whole multipart request into memory, which explains why large files ( > 100mb) cannot be uploaded.

Now I understand this is probably a limitation of how multi-part uploads work, and the only real work around would be to stream the multi-part data into a file, and then upload the file instead. However, browsers, python libraries, and many other frameworks are able to easily handle massive files when doing multi-part uploads. So I wonder if there's any better way to achieve this. Do we need chunked uploads? Is there an actual protocol to do this?

Lastly, not even RN's own fetch library can handle large multipart uploads. They all blow up with large files. It really seems like the only option to upload large files right now is to not use multi-part uploads, but rather upload plain files.

Thoughts?

Thanks.

TakaKeiji commented 5 years ago

rn-fetch-blob: 0.11.2

Hello all, this is mainly for iOS, but I have a feeling that Android has the same issue.

I'm wonder what would be the best way to do a multipart-form upload with large files. The idea is to upload key-value data, together with a file. However, iOS does not seem to allow large uploads with the current implementation when multi-part is used. More specifically, this line here (https://github.com/joltup/rn-fetch-blob/blob/master/ios/RNFetchBlobReqBuilder.m#L235) basically builds the whole multipart request into memory, which explains why large files ( > 100mb) cannot be uploaded.

Now I understand this is probably a limitation of how multi-part uploads work, and the only real work around would be to stream the multi-part data into a file, and then upload the file instead. However, browsers, python libraries, and many other frameworks are able to easily handle massive files when doing multi-part uploads. So I wonder if there's any better way to achieve this. Do we need chunked uploads? Is there an actual protocol to do this?

Lastly, not even RN's own fetch library can handle large multipart uploads. They all blow up with large files. It really seems like the only option to upload large files right now is to not use multi-part uploads, but rather upload plain files.

Thoughts?

Thanks.

Having similar issue trying to upload multiple files using multi-part, at some content size (depends on available memory) the request is sent incomplete (servers receive partial, broken or null data).

As you have noticed, building the request on memory seems to be the root of the problem, i'm very sure this is not a well designed behaviour to handle large amount of data, React native fetch and axios seems to have the same issue.

Finally checking the documentation, there is no clear or even if exists at all a way to perform a stream body upload that is very common procedure to deal with large request body, data and Session API or File Stream API seems to be only to handle data download only.