How to do a git clone to be used from memfs + questions about global variables

[nikitavoloboev]

I am trying to expose wasm-git as a raw interface similar to https://isomorphic-git.org/en/

I have this file:

https://github.com/nikitavoloboev/inlang/blob/libgit2/source-code/git-sdk/src/api/index.ts#L4

Trying to base it off https://github.com/petersalomonsen/javascriptmusic/blob/master/wasmaudioworklet/wasmgit/wasmgitworker.js

I get lots of TS issues as it doesn't know what FS means, it's a global variable, not sure where is it defined?

In the end I would essentially do a call similar to

    await raw.clone({
        fs: args.fs,
        http,
        dir: "/",
        corsProxy: clientSideEnv.VITE_GIT_REQUEST_PROXY_PATH,
        url: `https://${host}/${owner}/${repository}`,
    })

But with exposed wasm-git.

Also I see lg.callMain(['clone',http://localhost:5000/test.git, 'testrepo']); used in example but it doesn't seem to accept as argument a file system, is this possible to pass?

I essentially want to do a git clone with wasm-git to be used from https://www.npmjs.com/package/memfs

For FS issue, I at first thought I should do import FS = require('fs'); but wasmgitworker.js didn't have it so not sure.

Thanks for any help on this. ♥️

[petersalomonsen]

FS is a global object when loading an emscripten build. So when you have loaded lg2.js (using importScripts in a web worker) you should have FS on the global scope. FS is the File System API of emscripten: https://emscripten.org/docs/api_reference/Filesystem-API.html

For using in TypeScript you should just declare it as any, unless you find any good type definitions for it (I'm sure there are many out there).

[nikitavoloboev]

Thank you.

I think I am starting to figure out the web worker side of things.

https://github.com/nikitavoloboev/try/blob/main/solid/src/worker.js

Stuck trying to do a sparse clone with libgit.

I think I need something like

lg.callMain(["clone", "--depth=1", msg.data.repo])

But for some reason if I keep my worker simple like:

onmessage = function (e) {
  console.log("Worker: Message received from main script");
  const result = e.data[0] * e.data[1];
  if (isNaN(result)) {
    postMessage("Please write two numbers");
  } else {
    const workerResult = "Result: " + result;
    console.log("Worker: Posting message back to main script");
    postMessage(workerResult);
  }
};

All works.

But when I add code with libgitPromise as in wasmworker.js

All stops working. I am not sure what I am missing.

[nikitavoloboev]

What I got to so far: https://github.com/nikitavoloboev/try/blob/main/solid/src/worker.js

Added comments to where I am confused how to do some things. I manually moved the lg.js to repo (in my case I called it libgit2.js).

I am trying to understand how to add the github proxy thing so the clone works with CORS using new wasm-git syntax and not sure about some git clone commands, it's kind of hard to test. I call the worker from here.

Would truly appreciate any help on this, if I can make a clone into a virtual file system from github, I think I can try figure out how to make an edit to the file after amongst other things. ♥️

[petersalomonsen]

Hi,

I've reduced your WebWorker code to a minimum for cloning, in order to just hopefully make it possible for you to clone your repo. As you can see below, I'm loading lg2.js from unpkg.com (where they are prebuilt as a result of publishing to npm).

So this would be your minimum WebWorker code ( place it in a js file and load it using new Worker('workerfile.js') ).

This clones via localhost, where your proxy for github should be running ( code for the proxy is under the worker code ).

const WASM_GIT_BASE_URL = 'https://unpkg.com/wasm-git@0.0.9/';
var Module = {
  locateFile: function (s) {
    return WASM_GIT_BASE_URL + s;
  },
  'print': function (text) {
    console.log(text);
  },
  'printErr': function (text) {
    console.error(text);
  }
};

importScripts(WASM_GIT_BASE_URL+"lg2.js");

// from my understanding, this is needed to check the module got loaded properly
const libgitPromise = new Promise((resolve) => {
  Module.onRuntimeInitialized = () => {
    resolve(Module);
  }
});

(async () => {
  const libgit = await libgitPromise;

  FS.mkdir("/test");
  FS.mount(MEMFS, {}, "/test");
  FS.chdir("/test");

  libgit.callMain(["clone", "http://localhost:5002/nikitavoloboev/try", "try"]);
  FS.chdir("try");
})();

And here is the code for the proxy which you can run using nodejs:

const http = require('http');
const https = require('https');

function onRequest(request, response) {
    let path = request.url.substring(1);
    console.log(path);

    response.setHeader('Access-Control-Allow-Origin', '*');
    response.setHeader('Access-Control-Allow-Headers', '*');

    if( path.indexOf('git-upload') > -1 ||
        path.indexOf('git-receive') > -1) {  
        const options = {
            hostname: 'github.com',
            port: 443,
            path: request.url,
            method: request.method,
        };

        console.log(`Proxying ${options.method} request to ${options.hostname} with path ${options.path}`);
        const proxy = https.request(options, function (res) {
            res.pipe(response, {
            end: true
            });
        });

        request.pipe(proxy, {
            end: true
        });
    }
}

http.createServer(onRequest).listen(5002);

[nikitavoloboev]

Okay I tried that but hitting this issue:

My front end code here: https://github.com/nikitavoloboev/try/tree/main/solid

The code that creates the worker: https://github.com/nikitavoloboev/try/blob/main/solid/src/routes/web-workers.tsx

I added worker here: https://github.com/nikitavoloboev/try/blob/main/solid/src/workerfile.js

The issue is that the worker gets loaded but the issue is there because of this line:

https://github.com/nikitavoloboev/try/blob/main/solid/package.json#L8

The issue is that if I remove the line my solid framework breaks down as it expects things to be in modules.

Do you know if there is a es modules friendly alternative to importScripts?

https://stackoverflow.com/questions/61401475/why-is-type-module-in-package-json-file

Basically I am forced to use import syntax only. Can I load in wasm-git through import syntax and https://unpkg.com/wasm-git@0.0.9/ or should I perhaps download the file locally and try load it it in via import syntax?

[nikitavoloboev]

Perhaps something like

import * from 'https://unpkg.com/wasm-git@0.0.9/lg2.js'

var Module = {
  locateFile: function (s) {
    return WASM_GIT_BASE_URL + s
  },
  print: function (text) {

Not sure what should go in place of locateFile though and whether that import even works inside web worker environment.

[petersalomonsen]

It's very important that Module is declared before the import, otherwise it will not be applied. Do you have to create the worker with type: 'module' ( https://github.com/nikitavoloboev/try/blob/main/solid/src/routes/web-workers.tsx#L6 )? If you can skip this, then my example would work better.

[nikitavoloboev]

Ok indeed removing the type: module removes the error. Now everything loads with no error, however I don't know how to check if it worked.

I watched your demo of wasm-git and in there you show IndexedDB in dev tools but for me its empty.

[nikitavoloboev]

oh wait but I see it working

So I guess it loaded in, just its not stored in IndexedDB.

This is exciting. Thank you. 🎊

Okay I see why your demo had it in IndexedDB, FS.mount(IDBFS, {},/${currentRepoRootDir});. Not sure if I should use IDBFS or FS.mount(MEMFS, {}, "/test"). Is it better to store things in IndexedDB?

[nikitavoloboev]

Okay I see IDBFS sits on top of MEMFS actually, according to this answer.

Thanks a lot again.