Writing to specific folders not working?

[joshbrew]

Hi I've been using BrowserFS and recently translated to ZenFS and I am not clear at all on how this mounting system works.

So I want to do something with IndexedDB like

import {configure} from '@zenfs/core'
import * as fs from '@zenfs/core/promises'
import { IndexedDB } from '@zenfs/dom';

await configure({
    mounts:{
        '/data':IndexedDB
    }
});

await fs.writeFile('/data/list.csv','A,B,C');

And I get

Error: No such file or directory

???

[james-pre]

@joshbrew,

Thanks for submitting an issue. Could you please tell me what versions of @zenfs packages you are using? Thanks.

[joshbrew]

Just the latest 0.16.2, tested in Chrome. The directory exists if I used the exists() call while mkdir will hang or skip I'm not exactly sure what happens.

[joshbrew]

I should add that writing to just list.csv after mounting works (I guess it's just writing into /data anyway? The IDB snapshot is not transparent) but using the specified directory does not.

[james-pre]

What is the path of the error? (ErrnoError.path)

[joshbrew]

looks like it is showing up as just /list.csv when specifying as /data/list.csv

[james-pre]

When you specify a path, it gets resolved relative to the mount point of the FS. However, the error path should be corrected when an error is thrown. I will definitely be taking a deeper look into this.

I'm moving, so my investigation will take longer than usual.

[joshbrew]

How is that supposed to work with multiple mounts? I get the same error if I add like the /tmp InMemory DB as an additional mount. The most intuitive for me would be creating a folder to target the DB partition I want, then I specify files in that subfolder to write there, that's how I remember BFS working so then I could just create directories of a specific DB type as I go as I am creating folders dynamically to bucket datasets associated with different users, so the buckets are not accessed all at once in IDB.

[james-pre]

Every time you call a function, it solves the correct FileSystem, mount point, and path relative to the mount point. You can check out resolveFS in src/emulation/shared.ts.

[james-pre]

Hi @joshbrew,

Thanks for your patience. With @zenfs/core@0.16.2 and @zenfs/dom@0.2.14, I am unable to reproduce the issue with this code:

<script src="./node_modules/@zenfs/core/dist/browser.min.js"></script>
<script src="./node_modules/@zenfs/dom/dist/browser.min.js"></script>
<script>
    const { configure, promises: fs } = ZenFS;
    const { IndexedDB } = ZenFS_DOM;
</script>
<script type="module">
    await configure({
        mounts: {
            '/data': IndexedDB,
        },
    });

    await fs.writeFile('/data/list.csv', 'A,B,C');
    console.log(await fs.readFile('/data/list.csv', 'utf8'));
</script>

Console output is correct, A,B,C

[joshbrew]

I am bundling it with esbuild but same behavior with or without minification, here is my test application. Here is my test application. You can run it if you install the bundler system I use via npm i -g tinybuild which just wraps esbuild with a dev server.

taskmanager.zip

[joshbrew]

Actually in isolation it works, that zip I linked had some other issues though. I'm gonna assume it's on my end taskmanager.zip

[saoirse-iontach]

Maybe import type { Store } from '@zenfs/core/backends/store/store.js'; instead of from '@zenfs/core';

from '@zenfs/core' will use ZenFS globalExternals, where other will embed a custom copy.

This will result into two copy of the ErrnoError constructor, and will fail to match e instanceof ErrnoError.

[!CAUTION] The information in this comment is incorrect.

[james-pre]

Maybe import type { Store } from '@zenfs/core/backends/store/store.js'; instead of from '@zenfs/core';

from '@zenfs/core' will use ZenFS globalExternals, where other will embed a custom copy.

This will result into two copy of the ErrnoError constructor, and will fail to match e instanceof ErrnoError.

@zenfs/core maps to dist/index.js not dist/browser.min.js. ErrnoError is the same.

[joshbrew]

No the issue there was I didn't see I had pasted in the script tag twice and it was causing overrides. I still can't seem to work with the directory/subdirectory system in a way that makes any sense yet though. It seems like if I call mkdir it just creates a new StoreFS endpoint (which is still IDB in browser?) and I can't do subdirectories e.g. /data/dir, which in the case of IDB would just be a key in a flat mapped dictionary anyway but would tell us to send it there vs e.g. /localstorage/dir

I also get different behavior if I use the fs import versus importing the @zenfs/core/promises, where I just get undefineds when checking directories and stuff with fs while the async calls actually error and enforce things for me.

I've ended up just converting my whole wrapper file (which basically just documents the API) to native indexedDB calls and I get the exact behavior I want.

This is the hacky zenfs stuff I've tried but like createFolder errors in this usage. Other issue is I'm not accessing the same DB version each session so the application memory increases but I dont get the previous state on refresh.

//@ts-ignore
import { fs, configure, InMemory } from '@zenfs/core';
//@ts-ignore
import * as asyncFs from '@zenfs/core/promises' //has promise based calls 
//@ts-ignore
import { IndexedDB } from '@zenfs/dom';

const defaultDB = IndexedDB;

let mounted = {};

// Initialize file systems
export const initFS = async (
    mounts: { [key: string]: any } = { '/tmp': InMemory, '/data': defaultDB },
    oninit = (asyncFs) => { },
    onerror = (e) => { }
) => {

    let toMount = {};
    if(!mounted['/tmp']) mounts['/tmp'] = InMemory; 
    for(const key in mounts) {
        if(!mounted[key]) {
            toMount[key] = mounts[key];
        }
    }

    if(Object.keys(toMount).length === 0) {
        return;
    } 

    Object.assign(mounted, mounts);

    console.log(mounted, asyncFs);
    try {
        await configure({
            mounts: toMount
        });
        oninit(asyncFs);

        return true;
    } catch (e) {
        onerror(e);
        return false;
    }
};

export const createFolder = async (foldername: string) => {
    await asyncFs.mkdir(foldername);
    return true;
}
export const deleteFolder = async (foldername: string) => {
    await asyncFs.rmdir(foldername);
    return true;
}

//make sure it starts with the /
export const formatPath = (path: string, dir?: string) => {
    if (!path.startsWith('/')) path = '/' + path;
    if (dir) {
        if (!dir.startsWith('/')) dir = '/' + dir;
        path = dir + path;
    }
    return path;
}

//asyncFs.stat(path).then((stats)=>{ if(stats.isDirectory()) console.log("directory") }); //etc

export const checkDirInit = async (path: string, dbType = defaultDB as any) => {
    path = formatPath(path);
    let splt = path.split('/').filter(Boolean);

    if (splt.length < 1) throw `Bad Path: ${path}`; // no directory or invalid path

    const [dbName, folderName, subFolderName] = splt;

    // Check if the DB (first part) exists
    if (!await dirExists('/' + dbName)) {
        //console.trace(dbName, 'DNE, initDB', path);
        await initFS({ ['/' + dbName]: dbType });
        console.log(await asyncFs.readdir('/'+dbName));
        // createFolder('/'+dbName)
    } else {
        //console.log(dbName, 'DB exists', path);
    }

    // If there is a third part (sub-folder), check if the second part (folder) exists and create it if not
    if (subFolderName) {
        const folderPath = '/' + dbName + '/' + folderName;
        if (!await asyncFs.exists(folderName)) {
            //console.log(folderName, 'DNE, mkdir', folderPath);
            await createFolder(folderName);
        } else {
            //console.log(folderName, 'dir exists', folderPath);
        }
    }

    return path;
}

//for zenfs paths, start with the folder name, e.g. 'data/' like so, as it will make a (default indexedDB) type 
// Check if a path exists
export const exists = async (path = ''):Promise<boolean> => {
    path = await checkDirInit(path);
    return await new Promise(async (res,rej) => {
        try{
            let ex = await asyncFs.exists(path);
            res(ex);
        } catch(er) {
            res(false);
        }
    });
};

I had this all working before in BrowserFS but there seem to be different caveats and it's not obvious to me. wrapper.zip

[james-pre]

@joshbrew,

I still can't seem to work with the directory/subdirectory system in a way that makes any sense yet though. It seems like if I call mkdir it just creates a new StoreFS endpoint (which is still IDB in browser?) and I can't do subdirectories e.g. /data/dir, which in the case of IDB would just be a key in a flat mapped dictionary anyway but would tell us to send it there vs e.g. /localstorage/dir

StoreFS is used by the IndexedDB backend to avoid rewriting the entire FileSystem API.

If subdirectories are not working, it is most likely due to a misconfiguration. Make sure the mounts keys in the configuration use absolute paths.

I also get different behavior if I use the fs import versus importing the @zenfs/core/promises, where I just get undefineds when checking directories and stuff with fs while the async calls actually error and enforce things for me.

This is the same as the difference between importing node:fs and node:fs/promises, they are completely different APIs (the non-promises async API uses callbacks)

I apologize if I caused any confusion by using fs as the namespace import for the promises API.