search

scripting / Scripting-News

I'm starting to use GitHub for work on my blog. Why not? It's got good communication and collaboration tools. Why not hook it up to a blog?
121 stars 10 forks source link

Let's write a simple JavaScript pre-processor! ;-) #199

Open scripting opened 3 years ago

scripting commented 3 years ago

This has been a long-term interest, and now I actually want to create the code.

It's a pre-processor. It takes as input arbitrary JavaScript code as input.

  1. It runs it through a lexical scanner, and a parser, and out the other end comes a parse tree.

  2. We make a few changes to the parse tree.

  3. We serialize the parse tree back into JavaScript source text.

And that's it!

I've looked at several open source JS parser projects but haven't found code that does this simply to just do my stuff and get back to other projects. I don't want to get immersed in the processor. Hope this makes sense.

Any help much appreciated! :boom:

PS: I have experience with YACC and scanners. I've also written and maintained a scripting environment, so I'm fairly knowledgable about parsing, but am not familiar with any of the JS projects.

danmactough commented 3 years ago

There are quite a few JavaScript parsers that output an abstract syntax tree. You may be interested in exploring the superficial capabilities of a few of them using AST Explorer, which will render a full AST for any script you provide using one of several parsers:

image

For taking the AST and rendering it back to JavaScript, I've used escodegen, but it's been a while and there may be other/better options.

johnspurlock commented 3 years ago

You may want to check out Microsoft's TypeScript Compiler API.

It should take normal js files as input, and has a great in-memory AST implementation, which you can manipulate in Node.

I've been using it for a while now to do something like you're describing, but generating corresponding Java and Swift as output instead of js, since tsc already generates js : )

allenwb commented 3 years ago

Also, take a look at https://www.sweetjs.org

It is a Lisp inspired macro preprocessor for JavaScript

scripting commented 3 years ago

Thanks @danmactough, @johnspurlock and @allenwb for the tips.

I think I'm going to start with Dan's advice. I had tried this project a while back with Acorn. I'm going to spend a few hours to see what I can get working and will report back here.

amonks commented 3 years ago

re: your post about how to use AST explorer and why there is not a "button that says go":

2021-02-09 10 34 04

[I bet you already figured this out, but just in case 😉]

scripting commented 3 years ago

It took a couple of days of head scratching and trial and error, but I now have a skeletal JS preprocessor, I think/hope.

https://github.com/scripting/jsPreprocessor/blob/main/compile.js

Have a look, download, give it a try.

  1. I think it's general, meaning it would handle any JS code you throw at it.

  2. Disclaimer: I'm sure you can do this with builtin tools in one of the two packages. But I think it took me less time to write my own tree traverser than to figure out how to use theirs. Time was of the essence, I didn't want this feasibility study to turn into a monthlong project, as so many of my excursions do. ;-)

  3. Last time I worked in this area was in 1988 or 1989. But it's like riding a bicycle.

  4. The big breakthrough was realizing that it was a depth-first traversal.

scripting commented 3 years ago

Here we are two months into the development and productizing process, and I have to admit I blew myself away with this example script.

http://scripting.com/2021/04/09/141443.html?title=aDrummerExample

Make sure you get to the last part, about the Killer feature. ;-)