WIP Node.js port

[xkxx]

Hi Gravicappa,

This is a work in progress node.js port of Shen.

The following files are added:

• shen-node.js program entry
• runtime-node/lib.js runtime functions and hacks
• package.json required package file

  The following file is changed:

• .gitignore: ignore node_modules directory

  How to test:

• Clone the repo
• Install node.js on your platform. Make sure you can use npm and node commands
• npm install
• ./shen-node.js or node shen-node.js

  Bugs

• Sometimes entering a partially completed expression will cause endless recursive calls. This is caused by shen.js keeping calling repl_read_byte despite getting -1 responses. I think it's a problem with the shen.js file. To test this bug, type (str [enter] in the repl.
• Transpile/execute shen.js do not work. I haven't figured out how they work in shen.js yet.

Please test out the code and report any problems so I can fix them.

Regards, xkxx

[xkxx]

This port is still lack of transpile/execute support. I'd really appreciate if you can update shen-js to 9.1 spec and add support for compiling/executing shen strings. Or even better, support transpiling shen to readable javascript, similar to https://github.com/Gozala/wisp/. I'd be happy to help wherever needed.

[gravicappa]

I'm working on shen-js changes and on updating to latest Shen. What is transpiling?

[xkxx]

Thanks for the update. Transpile is to compile from one language to another. In most cases, these two words can be used interchangeably.

[gravicappa]

Translating to readable javascript is not possible: KL has features that are missing in js such as TCO, partial application, etc. Also, KL itself is not that readable. And I am not sure that this feature is useful anyway.

[xkxx]

A simple runtime library such as https://github.com/Gozala/js-tail-call can solve TCO issue quite easily. So for example, we would compile a tail call (define foo [X] -> (foo X)) as var foo = recur(function(X) {foo(x);};

Partial applications is not that difficult, it requires detection of partial applications and transforming them to things like return function(y) {return foo(x, y);}

The most tricky part, I think, would be lexical scoping, since javascript does not currently support it. The next version of Javascript, however, does support it via the let keyword, and is already available in Firefox, Chrome and Node.js (the latter two has to be enabled manually)

Overall, I would agree that transpiling to readable js would not be easy, but not impossible. If they can translate Clojure to Javascript, we can certainly translate 80% of Shen to Javascript pretty straight forward and the other 20% via some clever tricks. It can be a long-term goal for shen-js, I think.

As for the usefulness, there are three: performance, interoperability with JavaScript, and easy debug. The current shen-js has a major performance cost in evoking functions (which is what Lisp is). In order to evoke a function, you have to call shenjs_call first, which causes a 2x overhead comparing to normal Javascript. Also, since each variable is represented as an array, it causes a significant memory overhead, and to garbage collect these arrays bring forth performance impact too. These are all things to consider. The aforementioned problems also makes it harder working with other javascript code, and having shen_call and weird arrays all over the place is just inelegant. It makes debugging difficult too -- its always difficult to debug unreadable code, and all the anonymous functions that shen-js generates carry no semantics in the debugger, rendering stack traces useless.

[gravicappa]

First of all I have to mention that current shen-js is a pilot for reg-kl translator that is a part of translators set that aims to simplify porting of Shen into a wide set of languages. That means that some of translations of KL are not strictly required for js.

A simple runtime library such as https://github.com/Gozala/js-tail-call can solve TCO issue quite easily. So for example, we would compile a tail call (define foo [X] -> (foo X)) as var foo = recur(function(X) {foo(x);};

Currently TCO is done via trampolines that is similar to https://github.com/Gozala/js-tail-call.

Partial applications is not that difficult, it requires detection of partial applications and transforming them to things like return function(y) {return foo(x, y);}

This approach was used in previous versions of shen-js. And it wasn't the fastest way according to my benchmarks.

It can be a long-term goal for shen-js, I think.

Yes, it can be.

Also, since each variable is represented as an array, it causes a significant memory overhead, and to garbage collect these arrays bring forth performance impact too.

I didn't understand this part. Variables themselves are not represented as arrays. Symbols, conses, streams, closures do. I've tried and found no performance advantages of representing such entities as javascript objects.

The aforementioned problems also makes it harder working with other javascript code, and having shen_call and weird arrays all over the place is just inelegant.

Are "weird arrays" a function arguments array?

It makes debugging difficult too -- its always difficult to debug unreadable code …

If you need to debug generated code it means that translator is broken or incomplete.

… and all the anonymous functions that shen-js generates carry no semantics in the debugger, rendering stack traces useless.

I can't help noticing that implementing partial applications by function(a) {return function(b) {...}} or using trampolines for TCO introduce anonymous functions.

[xkxx]

This approach was used in previous versions of shen-js. And it wasn't the fastest way according to my benchmarks.

I'm curious to know what your current approach is.

I didn't understand this part. Variables themselves are not represented as arrays. Symbols, conses, streams, closures do. I've tried and found no performance advantages of representing such entities as javascript objects.

In that case I was mistaken in thinking that numbers/strings are represented as arrays. I would recommend using OOP for symbols, etc. Even if they don't have performance advantages, having them makes it a lot easier for debugging purposes. For example, console.info(array) will give you [Array] or [Shenjs_tag, ...] depending on runtime (Notice that the exact type is not revealed - just Shen_tag), while console.info(someObject) will give you the type of object and some other useful info esp. if you defined .toString() for the class.

Are "weird arrays" a function arguments array?

Yes. It is nearly impossible to manually call shenjs_call with correct arguments in plain js.

If you need to debug generated code it means that translator is broken or incomplete.

Or that there is a bug in your code that can only be revealed in runtime, which is fairly common.

I can't help noticing that implementing partial applications by function(a) {return function(b) {...}} or using trampolines for TCO introduce anonymous functions.

Take CoffeeScript (a language that transpiles to js) for instance, all functions/closures in the compiled code are named. This significantly benefits debugging.