`EmscriptenFS` and preloading

[pmp-p]

Hi, I have the following layout for running Pygame games apk directly on web

   // VM is emscripten Module, modularized or not

    VM.APK = "org.pygame.asteroids"
    var BFS = new BrowserFS.EmscriptenFS()

    const Buffer = BrowserFS.BFSRequire('buffer').Buffer;

    function apk_cb(e, apkfs){
        console.log(__FILE__,"APK", VM.APK,"received")
        window.document.title = VM.APK
        BrowserFS.FileSystem.InMemory.Create(
            function(e, memfs) {
                BrowserFS.FileSystem.OverlayFS.Create({"writable" :  memfs, "readable" : apkfs },
                    function(e, ovfs) {
                                BrowserFS.FileSystem.MountableFileSystem.Create({
                                    '/' : ovfs
                                    }, async function(e, mfs) {
                                        await BrowserFS.initialize(mfs);
                                        // BFS is now ready to use!
                                        await VM.FS.mount(BFS, {root: "/"}, "/data/data/" + VM.APK );
                                        await VM.FS.mkdir("/data/data/" + VM.APK + "/need-preload");
                                        VM.vfs = BFS
                                    })
                    }
                );

            }
        );
    }

    fetch(VM.APK + ".apk").then(function(response) {
        return response.arrayBuffer();
    }).then(function(zipData) {
        BrowserFS.FileSystem.ZipFS.Create({"zipData" : Buffer.from(zipData),"name":"apkfs"}, apk_cb)
    })

but emscripten_run_preload_plugins() cannot operate on Wasm/Image/Audio files located under /data/data//. error messages are all like
Image blob:http://localhost:8000/b5c7a41e-0824-4c8c-b1b2-672608468e73 could not be decoded `

browserfs 1.4.3

ref: https://emscripten.org/docs/api_reference/emscripten.h.html#c.emscripten_run_preload_plugins

[james-pre]

@pmp-p Have you tried with 2.0.0-beta?

[pmp-p]

nvm i only use preload plugin now for libraries and it was an emscripten bug https://github.com/emscripten-core/emscripten/pull/17956, forgot to close sorry

[pmp-p]

re-opening and quoting an emscripten expert on that "I guess FS.analyzePath in browserfs isn't setting .object.contents on the return value" if it can help to sched light.

Binary files that are expected to be preloaded by the emscripten plugins are truncated.

[james-pre]

@pmp-p

I'm not sure which parts of the BFS code you're referring to. analyzePath is not defined or used anywhere in the code.

If possible, could you create a minimally reproducible example with the latest code? Please note that the latest commit supports async/await on Create methods and configure. I recommend you use configure as it is much easier to use. For example:

import { configure, BFSRequire, ZipFS } from 'browserfs';

const mfs = await configure({
    fs: 'MountableFileSystem',
    options: {
        '/': {
            fs: 'OverlayFS',
            options: { 
                readable:  READABLE_FS,
                writable: { fs: 'InMemory' }
            }
        }
    }
});
const zipfs = await ZipFS.Create({ zipData: ZIP_FILE_BUFFER });
mfs.mount('/mnt/zip', zipfs);

[pmp-p]

So i built from git, using npm from emsdk with "npm run build" and used the un-minified browserfs.js + its .map found in ./dist/

[PyDK:wasm] /data/git/pygbag-TODO/BrowserFS $ npm run build && mv -v dist/browserfs.js* /data/git/pygbag/0.0/

> browserfs@2.0.0 build
> node scripts/build.mjs

Building for browser, unminified...
Built for browser, unminified.
Building for browser, minified...
Built for browser, minified.
Building for ESM, unminified...
Built for ESM, unminified.
Building for ESM, minified...
Built for ESM, minified.
Building for node...
Built for node.
renamed 'dist/browserfs.js' -> '/data/git/pygbag/0.0/browserfs.js'
renamed 'dist/browserfs.js.map' -> '/data/git/pygbag/0.0/browserfs.js.map'

---

let consider "test.apk" zip file which contains a text file "assets/main.py" eg https://v6p9d9t4.ssl.hwcdn.net/html/6836147/test.apk ( from an old working sample using older browserfs )

The goal is to mount in emscripten FS at path "/data/data/test" an overlayfs that read the zip file content and still allow to read/write/remove files and create/list/remove folders under "/data/data/test"

i tried without success ( folder and file structure is ok, reading zipped file content is corrupt)

var track_media
if (!vm) {
        vm={}

        // how is passed the FS object ???
        vm.BFS = new BrowserFS.EmscriptenFS()  // {FS:vm.FS}

        vm.BFS.Buffer = BrowserFS.BFSRequire('buffer').Buffer
        const data = await (await fetch('test.apk')).arrayBuffer()
        track_media = await vm.BFS.Buffer.from(data)
} else {
        track_media = MM[trackid].media
}

// how is passed the FS object ???
vm.BFS = new BrowserFS.EmscriptenFS()  // {FS:vm.FS}

vm.BFS.Buffer = BrowserFS.BFSRequire('buffer').Buffer

const data = await (await fetch('test.apk')).arrayBuffer()
const track_media = await vm.BFS.Buffer.from(data)

const zipfs = await BrowserFS.ZipFS.Create({
    zipData: track_media,
    "name": hint
})

const memfs = await BrowserFS.InMemory.Create();

const ovfs = await BrowserFS.OverlayFS.Create({
    writable : memfs,
    readable: zipfs
})

// this alone does not work ? why ??
// const mfs = await BrowserFS.MountableFileSystem.Create({"/" : ovfs})
// console.log( mfs )

const mfs = await BrowserFS.initialize( await BrowserFS.MountableFileSystem.Create({"/" : ovfs}) )

// where is the link beetween (BFSEmscriptenFS)vm.BFS and MountableFileSystem(mfs) ?
// vm.BFS.mount( ???????? )

const emfs = await vm.FS.mount(vm.BFS, {root: "/"}, "/data/data/test" );

needless to say i'm not very fond of javascript so i don't get some things in init sequence hence the above confused comments.

The working code (apart from corner case in preloading ) for older BFS

var bfs2 = false
if (!BrowserFS.InMemory) {
    console.warn(" ==================== BFS1 ===============")
    BrowserFS.InMemory = BrowserFS.FileSystem.InMemory
    BrowserFS.OverlayFS = BrowserFS.FileSystem.OverlayFS
    BrowserFS.MountableFileSystem = BrowserFS.FileSystem.MountableFileSystem
    BrowserFS.ZipFS = BrowserFS.FileSystem.ZipFS
} else {
    console.warn(" ==================== BFS2 ===============")
    bfs2 = true
}

            function apk_cb(e, apkfs){
                BrowserFS.InMemory.Create(
                    function(e, memfs) {
                        BrowserFS.OverlayFS.Create({"writable" :  memfs, "readable" : apkfs },
                            function(e, ovfs) {
                                BrowserFS.MountableFileSystem.Create({
                                    '/' : ovfs
                                    }, async function(e, mfs) {
                                        await BrowserFS.initialize(mfs);
                                        await vm.FS.mount(vm.BFS, {root: "/"}, "/data/data/test");
                                        //setTimeout(()=>{track.ready=true}, 0)
                                    })
                            }
                        );
                    }
                );
            }

            //await BrowserFS.FileSystem.ZipFS.Create(
            await BrowserFS.ZipFS.Create(
                {"zipData" : track_media, "name": "test.apk"},
                apk_cb
            )

i'm starting to think that code worked by accident and that my construction is broken.

[james-pre]

@pmp-p The non-fs polyfills have been removed (BFS is not a buffer/path/process polyfill, it is an FS polyfill). I recommend you use configure. For example:

import { configure, BFSRequire, EmscriptenFS  } from 'browserfs';

import ( Buffer } from 'buffer'; // polyfill. If running on node this is not needed.

const data = await (await fetch('test.apk')).arrayBuffer();

const track_media = await Buffer.from(data);

// assuming FS is from Emscripten

       

await configure({

    fs: 'MountableFileSystem',

    options: {

        '/': {

            fs: 'OverlayFS',

            options: {

                readable: { fs: 'ZipFS', options: { zipData: track_media, name: 'hint'  } },

                writable: { fs: 'InMemory' }

            }

        }

    }       

});

const fs =  new EmscriptenFS();

// ... set up emscripten ...

FS.mount(fs, { root: '/', }, '/data');

[pmp-p]

so

import { configure, BFSRequire, EmscriptenFS  } from 'browserfs';
import ( Buffer } from 'buffer'; // polyfill. If running on node this is not needed.

did not work for me ( syntax error, also i think import is not supported on some old/obsoleted mobile targets i have like safari/ios ). i load BrowserFS this way <script src="browserfs.js"></script> from html or from a script tag created from the main js file which has type=module but only when window.BrowserFS is undefined ( not the case during the test)

so i tried instead

const data = await (await fetch('test.apk')).arrayBuffer();
const track_media = await vm.BFS.Buffer.from(data);

// assuming FS is from Emscripten
await BrowserFS.configure({
    fs: 'MountableFileSystem',
    options: {
        '/': {
            fs: 'OverlayFS',
            options: {
                readable: { fs: 'ZipFS', options: { zipData: track_media, name: 'hint'  } },
                writable: { fs: 'InMemory' }
            }
        }
    }
});

vm.FS.mount(vm.BFS, { root: '/', }, '/data/data/test');

but same result zip content is garbled, folder structure is ok, file/folder creation works but reading newly created files back is garbled too.

[james-pre]

My bad... I had accidentally put a ( on the Buffer import. Here is the fixed version:

import { configure, BFSRequire, EmscriptenFS  } from 'browserfs';

import { Buffer } from 'buffer'; // polyfill. If running on node this is not needed.

const data = await (await fetch('test.apk')).arrayBuffer();

const track_media = await Buffer.from(data);

// assuming FS is from Emscripten

       

await configure({

    fs: 'MountableFileSystem',

    options: {

        '/': {

            fs: 'OverlayFS',

            options: {

                readable: { fs: 'ZipFS', options: { zipData: track_media, name: 'hint'  } },

                writable: { fs: 'InMemory' }

            }

        }

    }       

});

const fs =  new EmscriptenFS();

// ... set up emscripten ...

FS.mount(fs, { root: '/', }, '/data');

If you aren't using ESM, you will need to use BrowserFS from the browserfs.js, which defines It globally.

[pmp-p]

import { Buffer } from 'buffer'; But there is no buffer.js in dist. and i have no idea what ESM means

Maybe you are assuming all emscripten users have javascript knowledge, but that's not my case i only do C and python not even C++ ( emscripten is a C/C++ sdk ). Please don't take too much javscript shortcuts i'd really want to help solve that issue.

[james-pre]

You will have to install a buffer polyfill separately:

npm i buffer

[pmp-p]

Also i think i've made a shortcut too : the env testcase is not Node at all but only browser environnement ( emscripten target=web) and i'm testing on v8 not firefox

[james-pre]

I recommend you use a bundler (esbuild is fantastic) to bundle your code before running tests. This bundles dependencies which means you no longer need to worry about modules and imports. If you use esbuild, make sure you set bundle to true and platform to 'browser' in your esbuild config.

[james-pre]

Please use https://github.com/zen-fs/core/issues/22