[DISCUSSION] Optimize on mobile - replace `isomorphic-git` in favour of `wasm-git`

[joaquinrovira]

Firstly, I want to thank the maintainers and the community for this incredible plug-in.

The current implementation of obisidian-git is currently using isomorphic-git, a non-native Git implementation written purely in JavaScript. Although it is indeed a very interesting project, it is woefully slow even on a modestly modern phone (tested on Xiaomi POCO F3). Perhaps this issue could be solved by making use of a lower-level WASM implementation. We do not need to re-invent the wheel here as there are other projects that already implement this (e.g., petersalomonsen/wasm-git.

All this is assuming that whatever runtime being used in the Mobile apps actually support WASM. The following Rust plug-in template for Obsidian makes me think it is possible: trashhalo/obsidian-rust-plugin. If there is support by the maintainers/community I could volunteer to implement it. I am fairly familiar with Electron/Node and TypesScript, although my knowledge in regard to running WASM binaries is pretty slim.

[NicklasWallgren]

I had the same idea and started implementing a POC last week.

I'm planing to implement a simple git-sync-plugin using rust/wasm and wasm-git.

[Vinzent03]

I actually thought about giving wasm-git another try. I already tried it a year ago or so, but didn't succeed. The access to the file system could cause a problem, because I have to use the own provided by the Obsidian api. Another issue might be that I have to bundle it in the main.js.

[MeydanOzeri]

I tinkered a bit with wasm-git, it seems like it should be possible to use it, but modifications are needed.

Essentially 2 files are generated:

1. lg2.wasm which is the compiled c code of the library libgit2.
2. lg2.js which is the js code generated by emscripten to load and use the lg2.wasm at runtime.

From what I tested it doesn't work straight out of the box and modifications are needed either in the generated js file, or in emscripten build process (maybe both).

To get it to work with obsidian, the js file can be easily bundled into the main.js file and it will just load the wasm file at runtime and use it accordingly. The main challenge i see is changing the file system to use the obsidian file system, and i assume also the way http calls are made but not sure yet about that one.

So in my opinion to get it to work we need to change the code in lg2.js to use a custom made obsidian file system class having all the necessary methods that will be used. (similarly like it is now in isomorphic-git) Once it is built, it needs to be injected into the process before it tries to load the wasm file, and then load the wasm file through obsidian file system, I'm trying to accomplish it right now will update on my results.

[MeydanOzeri]

I tinkered a bit with wasm-git, it seems like it should be possible to use it, but modifications are needed.

Essentially 2 files are generated:

1. lg2.wasm which is the compiled c code of the library libgit2.
2. lg2.js which is the js code generated by emscripten to load and use the lg2.wasm at runtime.

From what I tested it doesn't work straight out of the box and modifications are needed either in the generated js file, or in emscripten build process (maybe both).

To get it to work with obsidian, the js file can be easily bundled into the main.js file and it will just load the wasm file at runtime and use it accordingly. The main challenge i see is changing the file system to use the obsidian file system, and i assume also the way http calls are made but not sure yet about that one.

So in my opinion to get it to work we need to change the code in lg2.js to use a custom made obsidian file system class having all the necessary methods that will be used. (similarly like it is now in isomorphic-git) Once it is built, it needs to be injected into the process before it tries to load the wasm file, and then load the wasm file through obsidian file system, I'm trying to accomplish it right now will update on my results.

Ok here is an update on the status, i managed to make it work and cloned a repo, both in pc and in IOS, i did it using wasm-git default virtual file system just to see that it works, now that i know its possible to run git like this all is left is to create and replace the file system with an obsidian file system and it should be fully functional (didn't test yet authentication) and can be integrated into the plugin as a replacement for isomorphic git.

[joaquinrovira]

I tinkered a bit with wasm-git, it seems like it should be possible to use it, but modifications are needed.

Essentially 2 files are generated:

1. lg2.wasm which is the compiled c code of the library libgit2.
2. lg2.js which is the js code generated by emscripten to load and use the lg2.wasm at runtime.

From what I tested it doesn't work straight out of the box and modifications are needed either in the generated js file, or in emscripten build process (maybe both).

To get it to work with obsidian, the js file can be easily bundled into the main.js file and it will just load the wasm file at runtime and use it accordingly. The main challenge i see is changing the file system to use the obsidian file system, and i assume also the way http calls are made but not sure yet about that one.

So in my opinion to get it to work we need to change the code in lg2.js to use a custom made obsidian file system class having all the necessary methods that will be used. (similarly like it is now in isomorphic-git) Once it is built, it needs to be injected into the process before it tries to load the wasm file, and then load the wasm file through obsidian file system, I'm trying to accomplish it right now will update on my results.

Ok here is an update on the status, i managed to make it work and cloned a repo, both in pc and in IOS, i did it using wasm-git default virtual file system just to see that it works, now that i know its possible to run git like this all is left is to create and replace the file system with an obsidian file system and it should be fully functional (didn't test yet authentication) and can be integrated into the plugin as a replacement for isomorphic git.

Incredible work, thank you for your efforts. Is there a fork where you are working on it? I would like to check it out and see if I can help.

[MeydanOzeri]

I tinkered a bit with wasm-git, it seems like it should be possible to use it, but modifications are needed. Essentially 2 files are generated:

1. lg2.wasm which is the compiled c code of the library libgit2.
2. lg2.js which is the js code generated by emscripten to load and use the lg2.wasm at runtime.

From what I tested it doesn't work straight out of the box and modifications are needed either in the generated js file, or in emscripten build process (maybe both). To get it to work with obsidian, the js file can be easily bundled into the main.js file and it will just load the wasm file at runtime and use it accordingly. The main challenge i see is changing the file system to use the obsidian file system, and i assume also the way http calls are made but not sure yet about that one. So in my opinion to get it to work we need to change the code in lg2.js to use a custom made obsidian file system class having all the necessary methods that will be used. (similarly like it is now in isomorphic-git) Once it is built, it needs to be injected into the process before it tries to load the wasm file, and then load the wasm file through obsidian file system, I'm trying to accomplish it right now will update on my results.

Ok here is an update on the status, i managed to make it work and cloned a repo, both in pc and in IOS, i did it using wasm-git default virtual file system just to see that it works, now that i know its possible to run git like this all is left is to create and replace the file system with an obsidian file system and it should be fully functional (didn't test yet authentication) and can be integrated into the plugin as a replacement for isomorphic git.

Incredible work, thank you for your efforts. Is there a fork where you are working on it? I would like to check it out and see if I can help.

I didn't fork, i created a test plugin just to play around with it, if you want here are the steps to use wasm-git in obsidian:

1. clone wasm-git
2. go to the folder emscriptenbuild
3. replace pre.js with:

   
   module.exports = async function (adapter, requestUrl, lg2Path) {

Module['instantiateWasm'] = async function(imports, successCallback) { const wasmBinary = await adapter.readBinary(lg2Path); const { instance } = await WebAssembly.instantiate(wasmBinary, imports); successCallback(instance); return instance.exports; };

if (!Module.print && !Module.printErr) { let capturedOutput = null; let capturedError = null; let quitStatus;

Module.print = (msg) => {
    if (capturedOutput !== null) {
        capturedOutput.push(msg);
    }
    console.log(msg);
}

Module.printErr = (msg) => {
    if (capturedError !== null) {
        capturedError.push(msg);
    }
    console.error(msg);
}

Module.quit = (status) => {
    quitStatus = status;
};

Module.callWithOutput = (args) => {
    capturedOutput = [];
    capturedError = [];
    quitStatus = null;

    Module.callMain(args);

    const ret = capturedOutput.join('\n');
    const err = capturedError.join('\n');
    capturedOutput = null;
    capturedError = null;

    if (!quitStatus) {
        return ret;
    } else {
        throw(quitStatus + ': ' + err);
    }
}

}

4. repalce post-async.js with: 
```js
/**
 * Javascript functions for emscripten http transport for nodejs and the browser NOT using a webworker
 * 
 * If you can't use a webworker, you can build Release-async or Debug-async versions of wasm-git
 * which use async transports, and can be run without a webworker. The lg2 release files are about
 * twice the size with this option, and your UI may be affected by doing git operations in the
 * main JavaScript thread.
 * 
 * This the non-webworker version (see also post.js)
 */

const emscriptenhttpconnections = {};
let httpConnectionNo = 0;

const chmod = FS.chmod;

FS.chmod = function(path, mode, dontFollow) { 
    if (mode === 0o100000 > 0) {
        // workaround for libgit2 calling chmod with only S_IFREG set (permisions 0000)
        // reason currently not known
        return chmod(path, mode, dontFollow);
    } else {
        return 0;
    }
};

if(ENVIRONMENT_IS_WEB) {
    Module.origCallMain = Module.callMain;
    Module.callMain = async (args) => {
        await Module.origCallMain(args);
        if (typeof Asyncify === 'object' && Asyncify.currData) {            
            await Asyncify.whenDone();
        }
    };

    Object.assign(Module, {
        emscriptenhttpconnect: async function(url, buffersize, method, headers) {
            method = method || "GET";
            let connection = {
                url: url,
                method: method,
                headers: headers,
                resultbufferpointer: 0,
                buffersize: buffersize,
                content: null
            };
            emscriptenhttpconnections[httpConnectionNo] = connection;
            return httpConnectionNo++;
        },
        emscriptenhttpwrite: function(connectionNo, buffer, length) {
            const connection = emscriptenhttpconnections[connectionNo];
            const buf = new Uint8Array(Module.HEAPU8.buffer, buffer, length).slice(0);
            connection.content = connection.content ? new Uint8Array([...connection.content, ...buf]) : buf;
        },
        emscriptenhttpread: async function(connectionNo, buffer, buffersize) {
            const connection = emscriptenhttpconnections[connectionNo];
            if (!connection.response) {
                const response = await requestUrl({
                    url: connection.url,
                    method: connection.method,
                    headers: connection.headers,
                    body: connection.content ? connection.content.buffer : null
                });
                connection.response = new Uint8Array(response.arrayBuffer);
            }
            let bytes_read = connection.response.length - connection.resultbufferpointer;
            if (bytes_read > buffersize) {
                bytes_read = buffersize;
            }
            const responseChunk = connection.response.slice(connection.resultbufferpointer, connection.resultbufferpointer + bytes_read);
            writeArrayToMemory(responseChunk, buffer);
            connection.resultbufferpointer += bytes_read;
            return bytes_read;
        },
        emscriptenhttpfree: function(connectionNo) {
            delete emscriptenhttpconnections[connectionNo];
        }
    });
} else if(ENVIRONMENT_IS_NODE) {
    const { Worker } = require('worker_threads');

    Object.assign(Module, {
        emscriptenhttpconnect: function(url, buffersize, method, headers) {
            const statusArray = new Int32Array(new SharedArrayBuffer(4));
            Atomics.store(statusArray, 0, method === 'POST' ? -1 : 0);

            const resultBuffer = new SharedArrayBuffer(buffersize);
            const resultArray = new Uint8Array(resultBuffer);
            const workerData =  {
                    statusArray: statusArray,
                    resultArray: resultArray,
                    url: url,
                    method: method ? method: 'GET',
                    headers: headers
            };  

            new Worker('(' + (function requestWorker() {
                const { workerData } = require('worker_threads');
                const req = require(workerData.url.indexOf('https') === 0 ? 'https' : 'http')
                              .request(workerData.url, {
                    headers: workerData.headers,
                    method: workerData.method
                }, (res) => {
                    res.on('data', chunk => {
                        const previousStatus = workerData.statusArray[0];
                        if(previousStatus !== 0) {
                            Atomics.wait(workerData.statusArray, 0, previousStatus);
                        }                    
                        workerData.resultArray.set(chunk);                    
                        Atomics.store(workerData.statusArray, 0, chunk.length);
                        Atomics.notify(workerData.statusArray, 0, 1);
                    });
                });        

                if(workerData.method === 'POST') {
                    while(workerData.statusArray[0] !== 0) {
                        Atomics.wait(workerData.statusArray, 0, -1);
                        const length = workerData.statusArray[0];
                        if(length === 0) {
                            break;
                        }
                        req.write(Buffer.from(workerData.resultArray.slice(0, length)));
                        Atomics.store(workerData.statusArray, 0, -1);
                        Atomics.notify(workerData.statusArray, 0, 1);
                    }
                }

                req.end();
            }).toString()+')()' , {
                eval: true,
                workerData: workerData
            }); 
            emscriptenhttpconnections[httpConnectionNo] = workerData;
            console.log('connected with method', workerData.method, 'to', workerData.url);
            return httpConnectionNo++;
        },
        emscriptenhttpwrite: function(connectionNo, buffer, length) {
            const connection = emscriptenhttpconnections[connectionNo];
            connection.resultArray.set(new Uint8Array(Module.HEAPU8.buffer,buffer,length));
            Atomics.store(connection.statusArray, 0, length);
            Atomics.notify(connection.statusArray, 0, 1);
            // Wait for write to finish
            Atomics.wait(connection.statusArray, 0, length);
        },
        emscriptenhttpread: function(connectionNo, buffer) { 
            const connection = emscriptenhttpconnections[connectionNo];

            if(connection.statusArray[0] === -1 && connection.method === 'POST') {
                // Stop writing
                Atomics.store(connection.statusArray, 0, 0);
                Atomics.notify(connection.statusArray, 0, 1);
            }
            Atomics.wait(connection.statusArray, 0, 0);
            const bytes_read = connection.statusArray[0];

            writeArrayToMemory(connection.resultArray.slice(0, bytes_read), buffer);

            //console.log('read with connectionNo', connectionNo, 'length', bytes_read, 'content',
            //        new TextDecoder('utf-8').decode(connection.resultArray.slice(0, bytes_read)));
            Atomics.store(connection.statusArray, 0, 0);
            Atomics.notify(connection.statusArray, 0, 1);

            return bytes_read;
        },
        emscriptenhttpfree: function(connectionNo) {
            delete emscriptenhttpconnections[connectionNo];
        }
    });
}

return Module;
};

5. follow instructions of building wasm-git in Release-async mode (instructions in .github/workflows/main.yml at line 52)
6. copy the result files lg2.js and lg2.wasm into your obsidian plugin root folder.
7. in the main.ts file use the lg2 like that:

   
   import { Plugin, requestUrl } from "obsidian";
   import lg2Initializer from "./lg2.js";

class GitTest extends Plugin { onload = async (): Promise => { const lg2 = await lg2Initializer(this.app.vault.adapter, requestUrl, 'path to your lg2.wasm'); lg2.onRuntimeInitialized = async () => { / Simple example of how it can be used / const FS = lg2.FS; // needs to be replaced with an obsidian filesystem const MEMFS = FS.filesystems.MEMFS; await FS.mkdir("/working"); await FS.mount(MEMFS, {}, "/working"); await FS.chdir("/working"); await FS.writeFile("/home/web_user/.gitconfig", "[user]\n" + "name = Test User\n" + "email = test@example.com"); await lg2.callMain(["clone", "https://github.com/torch2424/made-with-webassembly.git", "made-with-webassembly"); var testFile = await FS.writeFile("./made-with-webassembly/test.txt", "test") await lg2.callMain(["--git-dir=made-with-webassembly" ,'add', "test.txt"]); await lg2.callMain(["--git-dir=made-with-webassembly", "status"]); const files = await FS.readdir("made-with-webassembly"); console.log(files); }; }; }

[Vinzent03]

Thanks for your hard work. I've managed to run wasm-git on mobile and desktop, but I can't find a good way to use the obsidian file system. vault.adapter is async, but all file systems implementations I found you can mount are sync. The synchronized-promise package could be a solution, but that's not efficient and will probably block the UI. Do you know any way to solve this?

[MeydanOzeri]

I got stuck in the same place, from what i saw though there is a newly added file system to emscripten called wasmfs, which from discussions on it i saw that there is planned support for async file system, but, because there are no proper docs on it i have no idea how to enable it or if it is supported yet. I opened a discussion on it to get some help but as of right now no one replied: https://github.com/emscripten-core/emscripten/discussions/20048

Did you test the synchronized promise option ?

[Vinzent03]

Yeah I saw wasmfs as well. I'm currently implementing the fs for synchronized promise, so nothing to report yet. Thanks for creating the discussion! Let's hope someone responses and may help us.

[Vinzent03]

Update: I tried running synchronized promise, but it didn't work. From reading the README of the underlying package deasync it doesn't sound like that packages will run on mobile anyway. So I guess that way doesn't work, and we have to hope for an answer in your discussion...

[dbischof90]

This would also potentially solve the issue of mysteriously breaking repositories when used via iOS...

[fabhed]

First of all, thanks for the work put into this! 🚀

I simply want to highlight the user benefit of making the plugin more memory efficient by replacing isomorphic-git with a Wasm-based solution.

It would be a great improvement to allow for larger repos to be used with the plugin. I currently avoid cloning my assets folder of my notes on my Pixel 6 to not crash with an OutOfMemory error (repo size with asset folder is ~100 mb, and without it ~50mb).

[bardbess]

@MeydanOzeri do you need access to MEMFS? It appears mounting is possible without it (excuse me if I'm being ignorant here) which would revive the possibility of using Asyncify?

[quolpr]

If you need to call promise api from inside of wasm code you will need to use Asincify 🤔

I was exploring how it works in wa-sqlite recently, so maybe this may help you https://github.com/rhashimoto/wa-sqlite/blob/master/Makefile#L72

@Vinzent03 could you please make PR of you going work? 🙏

[Sparticuz]

Does the new async build help in wasm-git help? https://github.com/petersalomonsen/wasm-git/pull/89

[xVemu]

I think we all need this.