Wrong base64 encoding when retrieved back using Filesystem.readFile()

[tunecino]

Hi.

First of all thank you for creating this great library. I have a requirement of storing big audio files on which Capacitor FileSystem.writeFile() failed on android devices (crashes when the file is too big) that this library seems to handle very well.

Now the issue I am facing is when I try to load it back:

const { data } = await Filesystem.readFile({
    path,
    directory: Directory.Documents
  });

const blob2 = new Blob([data], contentType);
const blob3 = (await fetch(`data:${contentType};base64,${data}`)).blob();

/**
 *  blob2 -> Failed to construct 'Blob': cannot convert to dictionary.
 *  blob3 ->TypeError: Failed to fetch
 */

Filesystem.readFile() suppose to load a base64 encoded data which I need to pass to an audio player, but as it failed to read it, blob2 and blob3 here are just my attempts to re-construct the blob with the loaded base64 to ensure it is correct.

What is strange to me, is while both failed on web and iOS environments, blob3 worked fine on android and re-converted base64 from it did also work in the player.

I have no idea if it is related to the way this library is storing the blob, if it is related to the way Filesystem.readFile() is reading it, or if it is something else I am doing wrong.

I made a live sandbox to reproduce it here.

Any help is appreciated. Thank you.

[diachedelic]

Capacitor is capable of reading large files without encoding them as strings. All you need is the URI of the file on disk, which can be obtained with Filesystem.getUri.

write_blob({path, directory, blob}).then(function (uri) {
    audio_element.src = Capacitor.convertFileSrc(uri);
});

[tunecino]

Thank you very much.

That worked fine on Android. Unfortunately it did not work under web or iOs as I did only get a full path (starting with file:// ) on Android.

For web it looks like a known issue, but for some reasons Filesystem.getUri is returning an empty object on iOs and a dynamic path when using Filesystem.stat:

That is the related output on iOS of this code (got exact same output for stats and stats2):

return {
  path, // the uri returned by write_blob
  src: Capacitor.convertFileSrc(path),
  stats: await Filesystem.stat({
    path,
  }),
  stats2: await Filesystem.stat({
    directory: Directory.Documents, // tried also External and Data
    path: 'downloads/' + uuid,
  }),
}

[diachedelic]

This looks like a bug in Filesystem.getUri, which this plugin uses under the hood. Can you report it to the Capacitor maintainers?

[tunecino]

Yes I'll do. It is strange that I have only seen one (almost) similar case to mine here. I suspect it has something to do with my Nuxt app being generated as a PWA (that maybe makes it use some different storage?). I'll investigate it a bit more, make a mini repo to reproduce it and open an issue there. Thank you for the help.

[diachedelic]

I just realised, Capacitor.convertFileSrc expects an absolute path on iOS, not a file:// URL. So there should be no problem?

[tunecino]

A figured it out. It happens when you enable WKAppBoundDomains on iOS (see https://github.com/ionic-team/capacitor/issues/4721#issuecomment-873183407). Removing that key fixed it. Thanks.

[mrobst]

Hi - I raised an issue (here) and then saw this one which is almost a duplicate.

This library is great and working well for me on Android (thank you!). I can't find a way to get it to work on web though. At the moment according to https://github.com/ionic-team/capacitor/issues/3491 the ConvertFileSrc function doesn't work with the web so even though I can write a large file to local storage I can't find any way to read it back again. Using Filesystem.readFile does generate a string which looks like base64 but it doesn't work. I found out that if I paste this string into a repair utility (https://base64.guru/tools/repair) then it can be repaired and in fact the repair splits the string into two parts and the first part is correct. Maybe that could give a clue to what is happening? It would be even better to skip base64 conversion completely and avoid using Filesystem.readFile but I can't find a way to get at the file stored in web storage (indexDB) which looks like /DATA/xxxxxxxx - is there a way to read this directly as a file or blob? thanks for any pointers Marcus

[diachedelic]

Yes it is tricky. I only use the web platform for development of my app, so I was able to use the FileSystem API, which has poor browser support.

If you need to support all major browsers, you could try writing your Blob to IndexedDB, and then using the URL.createObjectURL function to access it as a URL.

[diachedelic]

Please report back if you find a solution, @mrobst .

[mrobst]

Hi - I've spent some more time looking into this and I can give a better description now of where I think the issue is....

When using capacitor-blob-writer in the browser the logic checks for Capacitor platform and triggers the write_file_via_bridge function which in turn calls append_blob. This function splits up the binary file into multiple base64 strings and calls Capacitor Filesystem.appendFile to save the blob into local storage.

I took the same file and first wrote it to local storage with Filesystem.writeFile then again with capacitor-blob-writer (which falls back to Filesystem.appendFile as it should). The resulting two files end up different with the latter being slightly larger in size (screenshot 1).

I then used Filesystem.readFile on both files. The one saved with Filesystem.writeFile returns valid base64. The one saved with append_blob/Filesystem.appendFile returns invalid base64. If I take this string and put it into an online repair utility ( https://base64.guru/tools/repair ) I get information that the file contains incorrect padding and multiple base64 values (screenshot 2).

My conclusion is that either Filesystem.writeFile behaves differently to Filesystem.appendFile (there is one line of code different between them in how the data / encoding is handled) or more probably the logic in append_blob here isn't quite chunking a blob into base64 correctly? Let me know if this is enough info to have a look at append_blob or if I should try and create a plnkr to reproduce the issue?

thanks! Marcus

[diachedelic]

Thanks for the digging. I've filed an issue with Capacitor here.

It should be possible to workaround the bug in the meantime.

[diachedelic]

I've attempted to fix the issue. Please test version 1.0.4 of the plugin and let me know how you go.

[tunecino]

Nice! Thanks for both of you. I'll try to test it too as soon as I get back to that project. My case was iOS running the web app instead of native (due to enabling WKAppBoundDomains), but my web code is tying on Filesystem.writeFile on web anyways as I failed to get it to work there, so I just did this kind of check to only use 'write_blob' on native devices where it worked great:

const isNative = Capacitor.isNativePlatform()
const content = isNative ? blob : await blobToBase64(blob)

[mrobst]

I've attempted to fix the issue. Please test version 1.0.4 of the plugin and let me know how you go.

Hi - thanks for the quick fix! First testing looks like its working OK. I'll let you know if I come across any issues. Hope the Capacitor team get chance to take a look at the underlying issue :-)