Turning JavaScript into a scripting language?

[scripting]

On Scripting News yesterday:

Suppose one were to contemplate a new scripting language that was minimally different from JavaScript, was implemented by a pre-processor (ie it generated JavaScript code), that could handle any JavaScript code or data structure, but -- allowed you to implement asynchronous functions that were called synchronously, both in the built-in functions and developer-defined functions, without using special syntax such as promises or await/async to invoke them. Has anyone attempted that? Delivered?

[scripting]

A superset of JavaScript, it's just a preprocessor, a very limited one

I was discussing this yesterday with Andrew Shell, he raised some concerns that I'd like to address here. I'll let him speak for himself as to what those concerns are.

The hypothetical language I'm interested in is a superset of JavaScript. You can use all features of JS, including promises, async/await, all the new syntax for constructing strings, etc.

Calling asynchronous functions exactly as if they were synchronous

It would have one main feature that JS does not have. The ability to define a function that is synchronous, even though it performs an asynchronous operation. The key point is this -- it is called in exactly the same way you would call a function that was perfectly synchronous. Two examples.

1. x = random (12, 200); //a function that does no I/O

2. filetext = fs.readFileSync ("config.json"); //a function that does I/O

As you can see, Node already has the kind of function I'm describing, so it isn't even a new concept. There obviously is something special about fs.readFileSync that allows it to suspend the process and make it wait for the response. I'd like to know how this is done in the interpreter, and armed with that information start to work on a general way to make that work.

I understand, from someone who has read the code, that this cannot be done outside of the Node kernel. Fine, so we'll do it with a preprocessor.

I am amazed how many programmers use fs.readFileSync, I very rarely do. I was told this would kill the performance of apps, but obviously it doesn't. :smile:

[scripting]

It's a preprocessor. I'm talking about a preprocessor that generates javascript. I want to put aside questions of how it's implemented for the time-being. I want to get any reaction to the idea of extending the language via a preprocessor. That's all.

[danmactough]

I want to get any reaction to the idea of extending the language via a preprocessor. That's all.

I think it's a great idea.

I'm very interested in the "how," though. Specifically, I want to know how you're thinking the preprocessor would know that a particular function call needs to be translated into a callback or Promise or async/await. I have a thought about that (although I took a brief run at it once and got stuck), so I'd love to hear what your plan is.

[tedchoward]

I like the idea.

There's lots of prior art for Javascript pre-processors. Babel is probably the most well-known. Most developers use tools like this to get early access to new language features, or for things like JSX.

As for your initial feature, hiding the async stuff, I want to see it in use. I think it's a good idea. It's definitely worth trying.

[scripting]

@danmactough, @tedchoward -- thanks for the encouragement.

Think about where this code will be running. I wasn't very clear about that...

Two places: LO2 and Electric Outliner (an Electron adaptation of LO2.

The scripts run in the context of the outliner, and do all the things Frontier's scriptable outliner can do. And additionally scripts can make HTTP requests and use websockets, and stuff can be layered on top of those capabilities. And in the Electron version, scripts running in the outliner do file system stuff.

Key point is this, we write this code, and we control how the functions are called from scriptland. Internally, we have a collection of glue scripts, written in system-oriented JS, meaning they exist in callback hell, but we know their names and when they're called we can generate special code that hides the details of synchronization from the script writer.

At first, only we'll be able to do this, but once we've burned the idea in, the next step will be to make the library extensible, so other devs can extend. But you'll need to know the secrets of making these hybrid objects to make it work. But script writers just call the code and let the underlying system take care of suspending and resuming processes.

Plenty of prior art here.

That's the concept.

[scripting]

A follow-up to this thread from November.

We have the basic runtime working. JavaScript as the basis for a scripting language, using a pre-processor.

The conceptual and development work was done by Andrew Shell, with a small assist from me, in getting the pre-processor up and running.

It was a short iterative process, I'll only describe the end result.

1. All the built-in verbs in the language return promises. Here's an example of what one looks like.

2. We insert the await keyword in front of every function call in the code we generate from the user's script code.

That's it.

As a result, something like this works.

dialog.alert (file.writeWholeFile ("tmp.txt", string.upper (file.readWholeFile ("tmp.txt"))))

And that was the goal.

Now I'm trying to rebuild the core of Frontier inside a copy of LO2. Not going as far as Frontier at first, I think that was one of our mistakes, not having something small and easy to grasp at first, and upgrade from there. I'm also writing the verb docs as I go, as carefully as I write the code. At some point soon, I'm going to want to invite former Frontier users into the loop to help review the verbs as they come online. I find I have questions that I wish I had some Frontier users around to help with.

[scripting]

A correspondent writes:

Have you done any performance profiling? Every await requires a full trip through the internal event loops. Might be too much overhead for hot synchronous internal subroutines.

No testing, but two notes.

1. this is for a scripting language. mostly quick and dirty things. or things that if they were too slow could be kernelized, a term we used in frontier for sucking things down into the C code, out of scripting land. 

2. what matters most here is the user's time. if something takes a minute every time they do it by hand, and it can be automated so it takes a second, that's the performance consideration that matters.

[tedchoward]

I like this because your goal is to optimize for user productivity/happiness, not machine efficiency.

So many language/framework decisions are made around machine efficiency, and user productivity suffers for it. Sometimes that is the right decision, but not always.

[scripting]

Thanks Ted. That was the philosophy in the development of Frontier. You'd be surprised how many things you'd think were too slow performed perfectly well in real life. Machines today are much faster than they were when we were learning how to program. Even a young dude like you! ;-)

[scripting]

BTW, here's a video that demonstrates a bit of performance, from a subjective point of view.

https://www.youtube.com/watch?v=QgCxiZo0164&feature=youtu.be

[hasseily]

Frontier was phenomenal as a complete scripting environment with an included structured datastore. The first iteration of our website ran on it. Rapid prototyping all the way to production.