node.js compatibility

[heapwolf]

If an os built with runtimeJS could use npm or npmd as a package manager it would inherit a significant ecosystem of highly usable code. To do so it would need some degree of node.js compatibility. This issue could be a discussion thread for the idea.

/cc @dominictarr @juliangruber @feross

[feross]

Is it this project's goal to build an entire OS, or just the kernel?

[heapwolf]

i think just the kernel, but the consideration here is: what the kernel exposes will determine what can be built on top of it. UPDATE: I updated the phrasing of the issue.

[feross]

Ah, the issue is much clearer with this new wording.

[iefserge]

I like this idea. It should be possible to create node.js compatible API on top of kernel services. But this is a long term goal. For now, we need to support at least one filesystem in R/W mode (ext2 for example).

I'm currently working on VFS layer (http://wiki.osdev.org/VFS) and initrd readonly filesystem. This FS should be automatically mounted to /initrd, so programs can access it. This is how it looks like, still in progress https://github.com/runtimejs/runtime/blob/master/initrd/system/vfs.js

Yeah, I think npm is a great package manager and we should be able to port it. But again, its long term goal.

[heapwolf]

awesome! fwiw i think this is the most exciting project since nodejs ;)

[thomasqbrady]

If you're starting from scratch, may I please request a more fault-tolerant file system? Something like BtrFS or ZFS? I really think the community will appreciate it later, and it would be easier to change now, right?

t h o m a s q b r a d y

On Jun 28, 2014, at 11:12 AM, Serge notifications@github.com wrote:

I like this idea. It should be possible to create node.js compatible API on top of kernel services. But this is a long term goal. For now, we need to support at least one filesystem in R/W mode (ext2 for example).

I'm currently working on VFS layer (http://wiki.osdev.org/VFS) and initrd readonly filesystem. This FS should be automatically mounted to /initrd, so programs can access it. This is how it looks like, still in progress https://github.com/runtimejs/runtime/blob/master/initrd/system/vfs.js

Yeah, I think npm is a great package manager and we should be able to port it. But again, its long term goal.

— Reply to this email directly or view it on GitHub.

[heapwolf]

@thomasqbrady probably better as a separete issue

[thomasqbrady]

Right, sorry.

t h o m a s q b r a d y

On Jun 28, 2014, at 11:33 AM, Paolo Fragomeni notifications@github.com wrote:

@thomasqbrady probably better as a different issue

— Reply to this email directly or view it on GitHub.

[crertel]

Erm, npm would be more of a userland thing, right?

Or are you proposing npm/bower/whatever-style build tools for generating the kernel itself?

[heapwolf]

@crertel no, npm would be a userspace thing, and any thing else like it. But in order for that to work there needs to be some consideration about what the kernel exposes.

[crertel]

Ah, that makes sense. It's my understanding though that that should only require basic file operations, right? Or am I missing something larger here?

[iefserge]

Well, all JavaScript code is userspace. This includes drivers and high-level kernel services too. It's microkernel architecture (http://en.wikipedia.org/wiki/Microkernel).

The main difference from classic microkernel is that system doesn't use hardware isolation, so no expensive hardware context switches. And IPC is very cheap (ArrayBuffers transferred between programs by passing a pointer internally, for example, no copy).

So we can put drivers in userspace too. Broken driver can't crash the whole system, and we can restart it any time. http://runtimejs.org/docs/arch.html

[groundwater]

This is pretty awesome.

Have you been able to bootstrap multiple copies of node in each isolate? I believe @zcbenz was working on a fork of node that played better with multi-isolates here https://github.com/zcbenz/cefode-node

I've been working on the "user land" side of this with node-os, and I would love to combine the two. I've been dreaming about the possibilities of using multi-isolates, and just doing away with the Unix process model, but you're actually doing it!

[iefserge]

It's possible to boot runtimejs in SMP mode, but this is pretty unstable and disabled by default. It runs 1 v8 isolate per core. No node API support at the moment, of course.

[groundwater]

I see two options here, each with its own challenges and benefits.

1. support the javascript-node apis
2. build the node source to run on runtime

The second approach requires compiling libuv which uses a pretty wide set of libc calls, including many system calls. For example, libuv uses a thread pool. There are no threads here.

Both approaches require a lot of work. I think implementing a compatible JS-api is more likely.

@iefserge does v8 only link against a small surface of libc? Can you point us to where you implemented those calls?

[iefserge]

Yes, JS api is the way to go. Node is quite different internally. Runtimejs currently uses limited set of musl libc library functions (http://www.musl-libc.org/). But this is a temporary solution, I'd like to remove this dependency at some point.

[mikeal]

Are you sure the best approach is to try to implement node's API?

You're definitely going to have an entirely different implementation of all the standard modules, including all the modules that use streams. If that is the case, will you really want to inherit all the known mistakes we've made with streams? We can, theoretically, do better. Ditto for the module system.

Also, W3C is doing a new stream API, lead by @domenic and while it is learning from the successes and failures of node's streams it is a totally different standard than node streams.

Compatibility, in the form of transpiling, should probably be a priority but I don't know if copying node's API entirely is the right answer.

[piranna]

I was not talking too much on this because I was not sure about the topic, but seems I was wrong thinking runtime.js was build using Node.js and in fact it's build using raw v8. I think it's a bad aproach, since this would means reimplement Node.js from scratch, or build an entirely new ecosystem, both things meaning a lot of work. I believe so runtime.js can be sucessful to build it using Node.js as basis or at least a fork of it focused on lightweight dependencies (something like a microNode, in the same way of MicroPython or PyMite, but using core Node.js code as basis instead of starting it from scratch). El 03/07/2014 01:50, "Mikeal Rogers" notifications@github.com escribió:

Are you sure the best approach is to try to implement node's API?

You're definitely going to have an entirely different implementation of all the standard modules, including all the modules that use streams. If that is the case, will you really want to inherit all the known mistakes we've made with streams? We can, theoretically, do better. Ditto for the module system.

Also, W3C is doing a new stream API, lead by @domenic https://github.com/domenic and while it is learning from the successes and failures of node's streams it is a totally different standard than node streams.

Compatibility, in the form of transpiling, should probably be a priority but I don't know if copying node's API entirely is the right answer.

— Reply to this email directly or view it on GitHub https://github.com/runtimejs/runtime/issues/3#issuecomment-47851643.

[RangerMauve]

The problem with node is that it replies on a lot of libraries which would need to be re-implemented for the kernel. Stuff like libuv would take a while to implement, and it might be easier to just work from v8, and add node apis after the core functionality of the kernel is working

[heapwolf]

Ok, so this thread is getting kind of chaotic. I think some important things to remember here are...

1. runtimejs is NOT an operating system, it's a kernel.
2. the things that it exposes will determine how an OS can be implemented on top of it.
3. a kernel wouldn't implement node.js API or streams or anything like that. those are user-space things.
4. yes node made a lot of mistakes. but by moving EVERYTHING (like streams) into user-space people can do whatever they want (wrong or right) and that will determine the popularity of the OS they build ;)

This thread was opened to determine what the kernel should expose so that something LIKE npm or modules from node's ecosystem could be run. This discussion should be way more low level IMO.

[dominictarr]

we javascripters are good at polyfils! as long as the right ideas are there, we can add a layer that makes it compatible with node. the first thing is doing right by runtime.js. designing for performance in the lower layer, and then designing for ease of use (which may mean node compat) in a higher layer. (look at how node has libuv then node.js on top)

[heapwolf]

This should be turned into a list of possible low level requirements and if those are practical and add value they can turn into api that will facilitate compatibility.

[mikeal]

IMO runtimejs should ignore node for a long as possible :) It's best if runtimejs is an amazing "plaform neutral" kernel. Once it's in good shape someone will come along, possibly me, and do whatever hacks are required to add a layer of compatibility on top that allows people to run Node modules :)

[heapwolf]

yes, that's the idea.

[piranna]

A platform generic kernel written in Javascript, or focused on Javascript apps? That's a huge difference in the design... El 31/07/2014 19:30, "Mikeal Rogers" notifications@github.com escribió:

IMO runtimejs should ignore node for a long as possible :) It's best if runtimejs is an amazing "plaform neutral" kernel. Once it's in good shape someone will come along, possibly me, and do whatever hacks are required to add a layer of compatibility on top that allows people to run Node modules :)

— Reply to this email directly or view it on GitHub https://github.com/runtimejs/runtime/issues/3#issuecomment-50791367.

[mikeal]

@piranna do you mean it's a huge difference from Node or a huge difference in terms of what you thought the intended design of runtimejs is?

[piranna]

About design of the kernel: if we intent to do a generic kernel, we'll need to add support for binaries, preemption, memory protection... things that for pure Javascript they are almost by free with the isolations... El 31/07/2014 19:44, "Mikeal Rogers" notifications@github.com escribió:

@piranna https://github.com/piranna do you mean it's a huge difference from Node or a huge difference in terms of what you thought the intended design of runtimejs is?

— Reply to this email directly or view it on GitHub https://github.com/runtimejs/runtime/issues/3#issuecomment-50793224.

[heapwolf]

I think what @mikeal meant was not generic kernel but less specific to the node platform.

[streamich]

@mikeal +1 for Node.js independent kernel