Uploaded files are not properly streamed with flutter for web

[technolion]

New Issue Checklist

• [x] I have searched for a similar issue in the project and found none

Issue Info

Info	Value
Platform Name	flutter for web
Platform Version	Flutter (Channel master, v1.16.3-pre.7, on Mac OS X 10.15.3 19D76, locale de-DE)
Dio Version	3.0.9
Repro rate	e.g. all the time (100%)

Issue Description and Steps

I am using dio to build a flutter web application that reads files from disk and uploads them to a server application. Unfortunately files are not streamed to the server, but read into memory completely and afterwards uploaded to the server. This is a big problem with large files, as the browser becomes slower and slower and sometimes even unresponsive. The problem only occurs with browser_adapter.dart. io_adapter.dart streams files to the server while the files are read from disk.

In order to be able to upload very large files, I am reading the files by using HTML5 File API and converting the read chunks to a stream:

//The passed File is a dart:html File
Stream<List<int>> _getStreamOfFile(File file) async* {
  FileReader reader = new FileReader();
  int chunksize = 1024 * 1024; // 1MB
  int start = 0;
  while(start < file.size) {
    int end = start+chunksize > file.size ? file.size : start+chunksize;
    Blob blob = file.slice(start, end);
    reader.readAsArrayBuffer(blob);
    await reader.onLoad.first;
    yield reader.result;
    start += chunksize;
  }
}

However the file is being read into memory completely before the XHR request is sent. Take a look at lines 93 to 99 of DIO's browser_adapter.dart:

    if (requestStream == null) {
      xhr.send();
    } else {
      requestStream
          .reduce((a, b) => Uint8List.fromList([...a, ...b]))
          .then(xhr.send);
    }

Calling reduce on the requestStream works through the complete file until the conversion is completed. Then the Future completes and xhr.send is being called. Also the above code is very CPU-intensive when running single-threaded as Javascript in a browser.

Would it be possible to use a StreamTransformer instead that transforms the requestStream as it is being read?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is still an issue, please make sure it is up to date and if so, add a comment that this is still an issue to keep it open. Thank you for your contributions.

[hauketoenjes]

Hi,

im currently running into the same problem with large files. I'm using the same approach by chunking the file. Have you come up with a solution to reduce the memory use?

[technolion]

The solution is not to use DIO for flutter web uploads. You should use dart.html.HttpRequest to upload a dart.html.File

This allows for streamed and fast upload within the browser.

[hauketoenjes]

Ok thank you for the info. A bit sad that this approach has no progress callback that is working properly..

[technolion]

To be fair, it's not the fault of DIO. The browser won't send progress callbacks, when uploading via XMLHttpRequest. We fixed this by writing a server side component that allows a client to fetch the progress information.

[hauketoenjes]

Yes of course this is not Dio's fault, was not meant to be a blame 😅.

Probably the best way of doing it with a server side component.

[karabanovbs]

Have the same issue.

Looks unoptimized:

     requestStream
          .reduce((a, b) => Uint8List.fromList([...a, ...b]))
          .then(xhr.send);

I think issue should be reopen.

[JohnGalt1717]

I really think that this should be addressed by the dart lang team.

You should be able to do this io.File.fromUri(Uri.parse(Url.createObjectUrl(html.File)));

I.e. since the web now allows full byte streaming, there's no reason why dart:io and Uri.parse shouldn't work to be able to create a dart:io file that can be manipulated (with limitations) to do file streaming etc.

[matheust3]

The solution is not to use DIO for flutter web uploads. You should use dart.html.HttpRequest to upload a dart.html.File

This allows for streamed and fast upload within the browser.

@technolion Would you have an example of how I could do this? My application needs to be able to grab files larger than 4GB :/