search

babel / babel

🐠 Babel is a compiler for writing next generation JavaScript.
https://babel.dev
MIT License
43.17k stars 5.64k forks source link

Architecture overhaul #883

Closed sebmck closed 9 years ago

sebmck commented 9 years ago

Ticks do not necessarily mean completed and may refer to deferration or reduction in scope.

sebmck commented 9 years ago

Please email me if you're interested in helping out.

sebmck commented 9 years ago

Unlocking this since I'd like some collaboration on this. Is anyone interested in maintaining a babel-cli package? I'm looking at spinning it out so someone else can "own" and maintain it.

/cc @matthewrobb @gsklee You guys both emailed me looking to help, thanks!

hawkrives commented 9 years ago

I'd be willing to help maintain a babel-cli – I don't know if I could take the entire thing on, but I'd love to help.

chicoxyzzy commented 9 years ago

count me in

gsklee commented 9 years ago

@sebmck I'm most interested in the transformer API and all kinds of internal optimizations. Let me know when I can be of help :)

chrisdwheatley commented 9 years ago

I'd be interested in helping out maintaining the CLI too.

neVERberleRfellerER commented 9 years ago

I would like to help with this overhaul. However, I won't have time till ~15th March.

chrisdwheatley commented 9 years ago

@sebmck What are your thoughts in terms of splitting out the CLI? Would be good to chat around what's required and see how other projects do it.

coreh commented 9 years ago

Some questions about this and retro-compatibility: Would the new external CLI inherit the existing command name, and accept the same arguments as babel(1)? Would the new package also be responsible for babel-node?

Also with this architecture overhaul, would we essentially be then on 5.x?

Naddiseo commented 9 years ago

I've been playing around with the pluggable parser idea at home. I have a general idea on how to implement one, but I'm currently struggling with what the interface to it would look like. And there is the issue of what happens when two pluggins conflict.

bvalosek commented 9 years ago

@sebmck can you elaborate on how you see type inference playing a part in babel?

The tern project had some great type inference for editors/IDEs but ES6 process on that is non-existent and will not be tackled anytime soon.

sebmck commented 9 years ago

@swirlycheetah

What are your thoughts in terms of splitting out the CLI? Would be good to chat around what's required and see how other projects do it.

Just abstract it out into another package and then alias the bin scripts in babel to them and log a deprecation message whenever you'd run them telling people to install/use babel-cli.

@coreh

Some questions about this and retro-compatibility: Would the new external CLI inherit the existing command name, and accept the same arguments as babel(1)?

Yep, ideally the babel package would install babel-cli for the remainder of 4.x for backwards compatibility and log a deprecation message.

Would the new package also be responsible for babel-node?

Yep.

Also with this architecture overhaul, would we essentially be then on 5.x?

Not unless it introduces breaking changes which is highly unlikely.

@Naddiseo

I've been playing around with the pluggable parser idea at home. I have a general idea on how to implement one, but I'm currently struggling with what the interface to it would look like. And there is the issue of what happens when two pluggins conflict.

Check out marijnh/acorn#212.

@bvalosek

can you elaborate on how you see type inference playing a part in babel?

Not at this time.

chocolateboy commented 9 years ago

Re: "Expressive node manipulation": I was investigating this a while back and spotted these:

chocolateboy commented 9 years ago

Re: "Pluggable [ AKA adaptive ] parser"

Probably not going to be integrated into Acorn any time soon, but the best approach I've seen to this is Katahdin.

sebmck commented 9 years ago

@chocolateboy

Re: "Expressive node manipulation": I was investigating this a while back and spotted these:

Yeah, I've already looked into them and I don't think they're an appropriate API by any means, probably something more akin to `ast-types.

Probably not going to be integrated into Acorn any time soon, but the best approach I've seen to this is Katahdin.

What are you basing this on?

chocolateboy commented 9 years ago

What are you basing this on?

Which part? :-)

Re: "not going to be integrated into Acorn any time soon": Katahdin is a language for writing languages. Acorn is a hand-written parser for a specific language. I can't see how the Katahdin approach could be integrated into Acorn without rewriting it from scratch.

IMO, making a parser/grammar extensible needs to be taken into consideration from the start rather than added as an afterthought. Obviously, I'd love to be mistaken about this :-)

The "best approach" bit is based on (what I consider to be) the ease of use (for developers). A high-level declarative approach is much easier to work with than low-level "hooks". Support for adaptive grammars is a bit of a holy grail, and the last promising thing I read along these lines seems to have been abandoned.

Also this:

technically there is research on adaptive/extensible grammars but that way leads to madness! -- tim disney

:-)

Naddiseo commented 9 years ago

@sebmck is there a design document/whiteboard/wiki I can share ideas on? I want some help breaking the problem down into smaller parts. The acorn PR seems a little restrictive, and I was going for a more general approach.

@chocolateboy, this isn't exactly "adaptive grammars" since we know ahead of source compilation time what the grammar will be, but it is definitely related. Not saying it's an easy problem, but likely easier than a grammar which can change itself at runtime (à la Perl).

chocolateboy commented 9 years ago

@Naddiseo: I think it depends on the scope of the "pluggable parser" proposal.

If the plan is to allow something like the mallet operator to be implemented as a third-party plugin (i.e. a self-contained syntax extension which doesn't interact adversely with other productions) without seeking approval from the Babel team, then I can see how that might be doable by adding hooks to Acorn.

If the plan is for everyone to stop reinventing the lexer/parser/transformer/emitter wheel and for people to be able to implement languages like CoffeeScript (rather than ES5+ extensions) on top of Babel, then I don't think that's a good fit for Acorn, or for anything else that's out there at the moment.

@sebmck is nothing if not pragmatic, so I expect the goal is the former, but I'd love to see Babel gravitate towards the latter.

Either way, I recommend taking a look at the Katahdin papers/language. There are a lot of good ideas there :-)

sebmck commented 9 years ago

@Naddiseo

is there a design document/whiteboard/wiki I can share ideas on? I want some help breaking the problem down into smaller parts.

Nope, only thing is this issue :stuck_out_tongue_winking_eye:

The acorn PR seems a little restrictive, and I was going for a more general approach.

If you feel as if it's restrictive though I'd recommend raising suggestions in marijnh/acorn#212 before it's too late as once it's merged, acorn-babel is going to use it and most likely all the parser modifications will be separated.

@chocolateboy

If the plan is to allow something like the mallet operator to be implemented as a third-party plugin (i.e. a self-contained syntax extension which doesn't interact adversely with other productions) without seeking approval from the Babel team, then I can see how that might be doable by adding hooks to Acorn.

Yeah, this is the sort of direction that I want.

If the plan is for everyone to stop reinventing the lexer/parser/transformer/emitter wheel and for people to be able to implement languages like CoffeeScript (rather than ES5+ extensions) on top of Babel, then I don't think that's a good fit for Acorn, or for anything else that's out there at the moment.

That's definently not what I'm proposing. I don't even know how this would work because supporting something like CoffeeScript would be such a shitfight that nothing would work together because the grammar would be so different.

chocolateboy commented 9 years ago

I don't even know how this would work

Katahdin

sebmck commented 9 years ago

@chocolateboy I only see additive syntax being practical.

chocolateboy commented 9 years ago

Fair enough, and I don't know C#, but what I got from spelunking the code/papers is:

yuchi commented 9 years ago

A new opponent has arrived! Did you have a look at sweetjs?

sebmck commented 9 years ago

@yuchi Yep, there was discussion about potential integration in #568.

Naddiseo commented 9 years ago

@sebmck Regards marijnh/acorn#212, after reading through the PR again it does look like it's probably flexible enough, but it only generates an AST, no CST

is there a design document/whiteboard/wiki I can share ideas on? I want some help breaking the problem down into smaller parts.

Nope, only thing is this issue :stuck_out_tongue_winking_eye:

Okay, I'll start a repo/wiki with my ideas so that this issue doesn't get polluted.

@chocolateboy

If the plan is for everyone to stop reinventing the lexer/parser/transformer/emitter wheel and for people to be able to implement languages like CoffeeScript (rather than ES5+ extensions) on top of Babel, then I don't think that's a good fit for Acorn, or for anything else that's out there at the moment.

Aren't they the same problem? If you solve the latter, then you've solved the former.

chocolateboy commented 9 years ago

@Naddiseo

Aren't they the same problem? If you solve the latter, then you've solved the former.

Implementation/effort-wise, I don't think providing a framework on top of which every AltJS language can be built is remotely the same as providing parser hooks so that something like the mallet operator can be implemented as a plugin, but -- like I say -- I'm eager to be proven wrong :-)

sebmck commented 9 years ago

Splitting this up into the follow issues: