FileWriter fails during FileRead step for large documents amidst a high volume of previous downloads on iOS

[connor-odoherty]

The problem as the user sees it:

1. User has long list of documents they need to download to iOS device (200+)
2. User starts the download, with each file downloaded in succession.
3. At the end of the download queue, they discover that one of the files fails (and its always one of two specific files that are 25MB+)
4. They retry the job (which only downloads the failed document) and it succeeds

What I'm seeing as a developer:

1. My app pulls down the document as a blob
2. The app calls this.file.writeFile(directoryPath, fileName, blob, {replace: true})
3. For the file that fails, I see the error pop out as

{"type":"error","bubbles":false,"cancelBubble":false,"cancelable":false,"lengthComputable":false,"loaded":0,"total":0,"target":{"fileName":"","length":0,"localURL":"cdvfile://localhost/persistent/downloaded-assets/9ce34f8a-6201-4023-9f5b-de6133bd5699/{{redacted}}","position":0,"readyState":2,"result":null,"error":{},"onwritestart":null,"onprogress":null,"onwriteend":null,"onabort":null}}

Which is not super helpful.

My debugging has led me to observe that the download fails in FileReader.js (which is called as part of the FileWriter step) with the line return this._realReader.readAsArrayBuffer(file); https://github.com/apache/cordova-plugin-file/blob/4a92bbbea755aa9e5bf8cfd160fb9da1cd3287cd/www/FileReader.js#L287

This is problematic because _realReader is set as this._realReader = origFileReader ? new origFileReader() : {};, and I have no idea how to access that for debugging.

To make matters worse, when I put a debugging statement here: https://github.com/apache/cordova-plugin-file/blob/4a92bbbea755aa9e5bf8cfd160fb9da1cd3287cd/www/FileWriter.js#L125, the error just prints out as an empty hash, telling me nothing.

Do you have any idea what might be going on here? Is it a memory issue on iOS? Does Cordova timeout over multiple repeated requests?

Something to note, the full list download works fine every time on the iOS xCode simulator. This is leading me to believe it might be a memory issue, but I'm not sure.

EDIT: I now see that the error is happening in the Ionic plugin (though this still might be an issue with the way Blobs are maintained in a Cordova environment)

I put some logging into the Ionic File plugin writeFileInChunks method. It became clear that the plugin, for whatever reason, error'd during a slice step.

  private writeFileInChunks(writer: FileWriter, file: Blob) {
    console.log('SIZE OF FILE AT START', file.size);
    const BLOCK_SIZE = 1024 * 1024;
    let writtenSize = 0;

    function writeNextChunk() {
      const size = Math.min(BLOCK_SIZE, file.size - writtenSize);
      console.log('CALCULATED SIZE:', size);
      const chunk = file.slice(writtenSize, writtenSize + size);
      console.log('SIZE OF CHUNK TO WRITE', chunk.size)
      writtenSize += size;
      writer.write(chunk);
    }

    return getPromise<any>((resolve, reject) => {
      writer.onerror = reject as (event: ProgressEvent) => void;
      writer.onwrite = () => {
        if (writtenSize < file.size) {
          writeNextChunk();
        } else {
          resolve();
        }
      };
      writeNextChunk();
    });
  }

The output for the failed document:

SIZE OF FILE AT START: 34012899
CALCULATED SIZE: 1048576
SIZE OF CHUNK TO WRITE: 0

On retry:

SIZE OF FILE AT START: 34012899
CALCULATED SIZE: 1048576
SIZE OF CHUNK TO WRITE: 1048576
CALCULATED SIZE: 1048576
SIZE OF CHUNK TO WRITE: 1048576
...
...
...
CALCULATED SIZE: 458467
SIZE OF CHUNK TO WRITE: 458467

So for whatever reason, after a large number of previous downloads, that file.slice step results in an empty/corrupted blob.

Any ideas on how to correct this?

Ran another test with some expanded logging:

  private writeFileInChunks(writer: FileWriter, file: Blob) {
    ...
    function writeNextChunk() {
      const size = Math.min(BLOCK_SIZE, file.size - writtenSize);
      console.log('CALCULATED SIZE:', size);
      console.log('WRITTEN SIZE', writtenSize);
      console.log('SUMS TO:', writtenSize + size)
      console.log('FILE SIZE BEFORE SLICE:', file.size);
      const chunk = file.slice(writtenSize, writtenSize + size);
      console.log('SIZE OF CHUNK TO WRITE', chunk.size);
      writtenSize += size;
      writer.write(chunk);
    }
    ...
    ...
  }

Output came to:

CALCULATED SIZE: 1048576
WRITTEN SIZE: 0
SUMS TO: 1048576
FILE SIZE BEFORE SLICE: 34012899
SIZE OF CHUNK TO WRITE 0

Further confirming the issue

[janpio]

What Ionic File plugin do you refer to? The Ionic Native wrapper for Cordova Plugin File?

[pynner]

@connor-odoherty any success since you posted?