Working together

[anbraten]

Hey, love to see that you basically did the same thing 😅

I've updated my README to use the same format as yours (hope that's fine) to check what features are identical and which are just in one of both. Looks like you did some interesting stuff on the isolation part with quickjs while my project has some additional shell stuff.

I am currently going to run each process in a worker similar to how webcontainer seems to be doing it as my tests with a wasm runtime where pretty slow while missing some features the browsers js runtime already provides. Have you tried that before?

For networking I've implemented a service worker to intercept all request from the iframe and forward them using postMessage, just intercepting request for the preview instead of the ide turned out to be way harder, why I think webcontainer is using a separate domain where they push the service worker and some files to somehow.

Would love to know what you found out.

[thecodacus]

hi. saw what you did with the shell. pretty long list of supported commands you have implemented. I tried to create a browser runtime. it quickly became a mess, so to switched to this alternative solution. I still have not figured out what to do about network yet. I just got the node importing module working.

[thecodacus]

oh I see you are using memfs, that can become troublesome i tried zenfs with in memory system.. the browser has lots of restriction so started maintaining files in MAP object

i was actually importing npm packages and extracting and writing them in the file system when i saw it breaking

[anbraten]

pretty long list of supported commands you have implemented.

Spent quite some time in that terminal already and got some parts like history, wget and unzip to help with debugging 😅

I still have not figured out what to do about network yet. I just got the node importing module working.

Networking seems tough. My plan is basically to polyfill the internal from node:net which is used by node:https createServer so I could try to run sth like vite.

Used memfs quite sucessfully, but switched to zenfs to use it more easily in workers. What was your issue with zenfs? The security headers for shared-array-buffers etc?

i was actually importing npm packages and extracting and writing them in the file system when i saw it breaking

Using wget and unzip worked for me with this for the beginning.

[thecodacus]

I am facing buffer allocation failed when trying to unzip https://registry.npmjs.org/npm/-/npm-10.2.4.tgz not sure if its due to zenfs. this is the reason I need to bundle npm a single js only containing the core parts

[anbraten]

Started to run node scripts in web-workers now, so each "node process" is using a worker. I am using a service worker to provide my virtual file system under localhost:5173/fs/.... That allows me to import js scripts and those can again import modules in the web-workers. That works impressively well already. My biggest difficulty is currently to get imports like import fs from 'node:fs'; redirected to my own mocks. I am getting errors like worker.ts:12 Module loading failed TypeError: Failed to resolve module specifier "node:fs". Relative references must start with either "/", "./", or "../". atm. My current approach is to use a regex replace changing import fs from 'node:fs'; to sth like import fs from '/fs/node:fs.js'; for all js files the service worker is serving. Seems pretty hacky, but couldn't find another way to patch the import resolution yet. Next plan is to write a mock for node:fs using a MessageChannel to the service workers fs and after that trying to adjust my own shell to only use node libs that I mock afterwards so it can run on node locally on my pc and inside the web-worker runtime.

Related files:

• https://github.com/anbraten/nano-web-ide/blob/main/packages/runtime/src/worker.ts
• https://github.com/anbraten/nano-web-ide/blob/main/packages/runtime/src/fsSnapshot.ts
• https://github.com/anbraten/nano-web-ide/blob/main/packages/runtime/sw.ts

How's your current progress?

[thecodacus]

wow nice.. I am trying a different approch. i have made the core container library which contains all the shell and node process. then I made a web worker file which imports thats whole core library then instantiate the container as global var

then I have one api library which has a worker bridge basically imports the worker's code and start the worker. now inside the worker i have the container running and I have setup some requests messages and paired response messages which can either spawn a process in container running inside the worker. and I have response message that worker sends back to worker bridge

and rest of the api package is just an abstraction of the of sending messages and receiving responses using the worker bridge so that it creates an abstraction layer and the enduser feels like they are directly interacting with the container

its working good so far. and as the container in running on the same worker as the js runtime its easier for me to just pass the file system as parameter then mock the imports