Consider utilizing the babel ecosystem

[AriaMinaei]

LiveScript is Beautiful™. Babel otoh, has a strong ecosystem. I think there is a great opportunity here for LiveScript to have both.

What if LiveScript generated a babel-compatible AST? It would still have its own separate parser, but after parsing, it would produce an initial AST, apply a few LiveScript-specific transformations on it to preserve some semantics, and then allow all the existing babel transformers to take care of the rest.

This would allow LiveScript users to take advantage of most of the great tools that have been built around babel, namely:

• ESLint: ESLint already allows custom parsers. babel-eslint converts babel's ASTs to that of ESLint's, which allows babel users to keep using babel's new features along with ESLint. LiveScript can do that too, and with a babel-compatible AST, that would be easy. We'll get a linter for free. There are also tools built around ESLint. People are already familiar with it. Why re-invent the wheel?
• react-transform-hmr: Hot realoading for react components. While there is a webpack loader that works with LiveScript, that loader is deprecated, and to use the new one, you have to use babel. If you are using HMR, I'm sure you can't imagine going back :)
• Relay: Works exclusively with a babel transformer.
• Babel Typecheck: Understands flow type annotations and adds runtime type guards.
• Other useful transforms and tools: react-transform-catch-errors, React constant elements transformer, React display name transformer, Documentation.js, astexplorer, ...

Note that the benefit doesn't just come from babel's existing tools. It also comes from all the activity that's been done around babel. For example, there is a lot of work put into writing this handbook which explains to beginners like me, how to write custom transformers for babel. It's easy and approachable. There is no need to redo this work all over again to write the same thing for LiveScript.

The killer feature for me, would be type checking with flow (though LiveScript would have to support parsing flow-style type annotations). If flow allowed custom parsers (we should open an issue for that), then we could (theoretically) use all of flow's type checking and intellisense features with LiveScript. This solution (again, theoretically), would be more ergonomic than the approach proposed here. The build process would be faster (LiveScript won't need to emit intermediate JS files with type annotations for them to later be processed by flow), and we'd get live feedback and intellisense inside .ls files, rather than in the compiled .js files.

All of this, I imagine, would make LiveScript both more powerful, and more approachable. LiveScript, imho, is serving a much smaller number of users than its real niche. That's not because of its syntax. It's because of its ecosystem. The need for a clean syntax, and type checking, aren't mutually exclusive. But the latter gets more important as the project grows.

Having taught UI programming to quite a few programmers, in CS and LS, I know that the initial resistance to the syntax quickly wears off, once people realize how much more productive they become when using it. And having used LS in production (currently running a 14K SLOC app by a team of 4), I know how beneficial it is, especially for growing projects. I wish more people knew this and took advantage of it. And I think there is an opportunity here to enable just that.

[vendethiel]

you might be interested in #803 & #811

[AriaMinaei]

Yes. Seeing #803 and #811 prompted me to open this issue. I figured since people are thinking about 2.0, it might be a good time to consider an architectural upgrade.

I'd love to know what you think of this :)

[heavyk]

I'm seriously +1 on this one, personally... there are millions of plugins and all kinds of ast transformations could be leveraged from js for optimization and stuff like that.

[waynedpj]

:+1: new to JS, love LS but did not know much about Babel until this issue. seems like a great fit for the next LS. thanks @AriaMinaei

[darky]

:+1: for this idea, I think too about it. Convert LS to Babel AST seems very hard and almost all rewriting. But more easer situation, when LS will convert in ES201(5|6|7...) code and then via babel.transform to ES5 compatible js. babel.transform also return map property, which can be used for properly generate Source Maps from LS to Babel output js.

[chrisfls]

:+1: on this. really hard and time consuming, but great reward :smiley:

[auvipy]

very big +1. would love to get involved and contribute!!!

[igl]

What's a babel AST?

Had a look at babylon and i don't see how you would hook-up into the babel eco-system other than compiling to proper es6 and let babel do it all over again.

[vendethiel]

Babel's Abstract Syntax Tree. I guess they don't use the SpiderMonkey AST

[AriaMinaei]

What's a babel AST?

Maybe I could use more accurate terminology here, but by "babel AST," I mean the data structure that's produced during babel's compile pipeline after lexing and parsing, and follows this spec (itself based on ESTree Spec), and it's compatible with babel's transforms.

Had a look at babylon and i don't see how you would hook-up into the babel eco-system other than compiling to proper es6 and let babel do it all over again.

Not sure what you mean by that. Obviously the different stages of babel aren't so tightly coupled as to disallow replacing one stage with a different implementation of it. If I understand correctly, we can replace babel's parser with LiveScript's, if the generated AST was compatible with babel's AST spec.

[AriaMinaei]

Babel's Abstract Syntax Tree. I guess they don't use the SpiderMonkey AST

Supposedly, it's based on ESTree, aka the SpiderMonkey AST.

[vendethiel]

Okay, thanks. Anyway, this project is an insane-scale undertaking, and I don't think I'm gonna have the time for anything like this anytime soon. I agree what we currently have (generating loc-annotated JS string fragments to be embbeded in other string fragments) is sub-par, but changing it now would mean a massive overhaul of the compiler.

[igl]

I see, there is a spec. I don't know if it's worth calling it it's own AST format: ESTree

ESLint totally works for me without babel-eslint. I am not using much esoteric es7 features other than object-spread though.

If I understand correctly, we can replace babel's parser with LiveScript's, if the generated AST was compatible with babel's AST spec.

I would say that's 90% of the complexity and I did not see a option to simply replace the parser in the API of babel.

[AriaMinaei]

@vendethiel I understand. It is a big undertaking.

I've started doing some experiments, just to get an idea of how big of a challenge it actually is. As predicted, lexer.ls and grammar.ls are completely salvageable, but ast.ls isn't.

The good news is that babel itself can handle a lot of the complexity. It already has transforms for many LS features, such as default parameters, rest parameters, destructuring, splats, constants, etc. It also has nice facilities to handle scope. So, as long as LS produces an AST that babel accepts, these features will mostly work out of the box.

The not-so-good news, is that other features such as everythingIsAnExpression, pipes, currying, and others, need to be reimplemented as babel transforms. That's probably gonna take a lot of work, but it has an upside, and that is, it makes LiveScript's compile pipeline more approachable for the average JS developer.

So far in this experiment, I've been able to produce some JS produced by babel from an LS input. The next step would be to implement some of LS` semantics as a series of babel transforms. But before doing that, I want to try and see if I can get ESLint working with the new pipeline. That would be pretty cool.

I'll update this thread as I make progress. But since I can only spend a couple of hours a week on this, expect slow updates.

[AriaMinaei]

@igl: I would say that's 90% of the complexity and I did not see a option to simply replace the parser in the API of babel.

That really is not a big deal. Babel could provide an option to use a custom parser, ala ESLint. There are other ways to handle this too.

The real challenge is enforcing LS' semantic rules as babel transforms.

[AriaMinaei]

So, as a proof-of-concept, I was able to write a little converter from LiveScript AST to Babel AST. For now, it only supports function declarations and simple expressions, but we can most likely expand it to support all of LiveScript's features.

I also connected the converter to babel-eslint, and now I get interactive linting hints in sublime-text (and probably other editors too):

[vendethiel]

that looks pretty good. One of my worries: does it support source maps?

[AriaMinaei]

@vendethiel It does, and it works perfectly! All we need to do is to convert LS' column/line numbers to babel's SourceLocation objects, like this:

loc:
  start:
    column: complexNode.first_column
    line: complexNode.first_line
  end:
    column: complexNode.last_column
    line: complexNode.last_line

Here, complexNode is just a reference to an LS AST node.

If people are interested, I'll put my fork online (just needs a bit of cleaning up).

[vendethiel]

I'd be very interested, yes. We might actually pull that off, and using this opportunity to clean up our ASTs nodes, like the good work @lydell is doing on coffeescript :) (props to him)

[waynedpj]

@AriaMinaei very cool, thanks!

[AriaMinaei]

Great! I will upload them as soon as I can, definitely before the start of next week :)

[AriaMinaei]

Okay, I made a small PR in #862.

[danielo515]

What is the status of this? As the original propose said, targeting babel AST is an huge advantage. Just think about the massive ecosystem you will immediately gain, for free. Not to mention how easy is to write a babel plugin to make it do literally whatever you want.

Please, please...

[danielo515]

Happy Christmas everyone. From time to time (twice a year usually) I wish that livescript continues evolving and starts using babel ecosystem. How do you feel about this ?

[determin1st]

babel is just back-compat syntax generator, right. what's the point?

maybe, i just dont understand.. if you do LS => JS => Babel => JS, it doesnt work?

[rhendric]

If someone writes a Babel back-end, I'd be happy to work with that person to land it here. dk00/livescript-next looked like it might be that project, but it's starting to look a little stale. Maybe someone could use it as a starting point, though?

[danielo515]

if you do LS => JS => Babel => JS, it doesnt work?

That will introduce one extra transpilation step, and hence one more place for bugs to hide and non intentional code to be generated. Second, there are several tools that work on an ast level, and there are several transformations that will not be possible after livescript has taken some metadata after compiilation. At first, I though the same as you, but the more advanced tings you try to do with babel, the more unlikely is that this setup will work as expected.

[danielo515]

If someone writes a Babel back-end, I'd be happy to work with that person to land it here. dk00/livescript-next looked like it might be that project, but it's starting to look a little stale. Maybe someone could use it as a starting point, though?

I been using livescript-next for a couple of months with great success. I looked into his code, and it is a bit hard to understand to my taste. Part because I only wrote a couple of babel plugins and a simple DSL, so I don't have huge experience creating compilers and part because I'm not very familiar with advanced usage of LS, but that may change.