Support TypeScript/Flow type annotations

dead-claudia commented 9 years ago

Edit: Missed a difference...

This has come up before (#393), but I feel it could get resurrected again, since the syntax appears to be stabilizing. TypeScript would be much easier to use with macros, and Flow has kept a very TypeScript-like syntax as well. Babel supports Flow type annotations officially, and keeps them in its internal AST for plugins to manipulate. People have attempted to use SweetJS as-is with them, with varying degrees of success (generally little). And Closure Compiler also appears to be going in the direction of allowing TypeScript annotations as well. Could this use case be supported?

The type syntax overlaps tremendously between the two, and they only really differ in a few areas, mostly in semantics:

TypeScript's {} vs Flow's mixed
Their type-checking behavior is different with Object
Flow's types are non-nullable by default, unlike TypeScript's.
Flow tends to infer a lot more types than TypeScript.

As for pure syntax, you could parse them identically except for TypeScript's new () => T vs Flow's Class<T>.

vendethiel commented 9 years ago

Could this be fixed with a custom readtable?

natefaubion commented 9 years ago

Could this be fixed with a custom readtable?

No, because its basically impossible to tell from a read whether : is going to be for something like an object property, a label, or a type signature. This is something that has to be built into the expander so that it can recognize all these forms for purposes of hygiene. It also means macros that potentially match on binding forms (function args, vars, etc) have to know the type syntax otherwise they won't work.

dead-claudia commented 9 years ago

There are also other productions that would need to come in shortly thereafter, but this bug more or less blocks those.

Type aliases
Interfaces
Generic classes

All three have the same syntax between Flow and TypeScript (mod the nit with constructors).

disnet commented 9 years ago

Yeah, I'm definitely open to this.

I think we would want to make sure our handling of the type language is generic enough to deal with both TS/Flow etc.

My main concern would be the added language complexity macro authors who want to handle functions/vars would need to deal with as @natefaubion pointed out.

kkirby commented 8 years ago

Any update on this?

disnet commented 8 years ago

Work is continuing on the redesign (#485). Once that lands it will make sense to think about adding type annotation support.

elibarzilay commented 8 years ago

+1, imported from Microsoft/TypeScript#4892

KiaraGrouwstra commented 8 years ago

disnet on gitter: "as long as your macro forms “look” like normal ES6 code you can pass it through TS/babel first just fine".

Trying out a ts -> babel -> sweet webpack workflow I still encountered a transpilation issue, but otherwise this option may be of interest for in as far we'd manage to cope with this limitation on external macro 'looks'.

elibarzilay commented 8 years ago

@disnet, Sounds like you're talking about macros that mostly look like function applications, which would be the most boring ones. (Ie, macros where you're playing with control flow, which are not too relevant now that the syntactic weight of wrapping stuff in thunks is tiny.)

amir-arad commented 8 years ago

We've released an alpha version of our TS visitors framework called tspoon . Tspoon should enable ts_parse-> sweet -> ts_transpile -> ... meaning you can run sweet as pre-transpile macros, and even disable specific transpiler warnings and the like.

elibarzilay commented 8 years ago

@amir-arad, IIUC, that project provides a hook into TS somewhere between parsing the input and producing the output. If that's correct, then it's kind of an option to implementing some macros on top of TS, so in theory it could be used to implement sweet.js for TS.

But I'm sure that there's lots of things that sweet is using that would be anywhere from inconvenient to impossible to do with the TS representation, which is why I think that the right way to go would be to extend sweet.js to recognize TS syntax and spit it back out (for TS to process).

KiaraGrouwstra commented 8 years ago

I'm sure that there's lots of things that sweet is using that would be anywhere from inconvenient to impossible to do with the TS representation

Would you mind elaborating on this? Wasn't TS just a superset of ES6 anyway?

I think that the right way to go would be to extend sweet.js to recognize TS syntax and spit it back out (for TS to process).

Well, if one were to count outstanding ES proposals (e.g. decorators, type annotations, though I forgot which of my Babel plugins added that), I believe TS actually does not really add much syntax over what has been proposed for ES already -- the extent of 'not much', AFAIK, being decorators on parameters.

So in that sense, might those two roads not be closer to being one and the same?

dead-claudia commented 8 years ago

@tycho01

Would you mind elaborating on this? Wasn't TS just a superset of ES6 anyway?

I know I wasn't the one you were asking, but I think Sparkler is a very good example of this. The TypeScript parser isn't particularly extensible, either (it's implemented in a giant closure).

Well, if one were to count outstanding ES proposals (e.g. decorators, type annotations, though I forgot which of my Babel plugins added that), I believe TS actually does not really add much syntax over what has been proposed for ES already -- the extent of 'not much', AFAIK, being decorators on parameters.

So in that sense, might those two roads not be closer to being one and the same?

You're correct in that TS doesn't add much syntax that isn't already either in or proposed for ES now. But type annotations are probably harder to parse than even the rest of the language.

elibarzilay commented 8 years ago

[@tycho01]

I'm sure that there's lots of things that sweet is using that would be anywhere from inconvenient to impossible to do with the TS representation

Would you mind elaborating on this? Wasn't TS just a superset of ES6 anyway?

I'm talking about the technical level. The kind of things that a good macro expander needs from a representation of syntax can be very demanding on one hand, and subtle on the other. (And sweet.js is one of the very few rare cases of a good macro system.) I doubt that a generic exposure of syntax nodes would be as full as needed (but TBH I didn't look too deeply).

[@isiahmeadows]

The TypeScript parser isn't particularly extensible, either (it's implemented in a giant closure).

Right -- that when I figured that for the TS people to add macros would be wrong from all kinds of aspects. It requires certain skills that people who really care about syntax enough have (to know what hygiene means), but they probably don't. It's a whole layer of complexity that actually has very little with the goals of TS. And like Flow, it's coming from a neighborhood of people who don't care much about such things anyway.

And then I realized that this can be just like the Typed Racket case, where the typed language doesn't do anything interesting at the syntax level (besides typechecking, of course), and instead it just expands the code completely before it starts. So especially since the TS and Flow worlds have settled on a very similar syntax, this should really go as an extension of sweet.js which could then be consumed by one of these.

And in the typed racket case, there was some pressure from a few people to have some type information available to the expander, so it's possible to use that information to expand macros in different ways based on it. In that case there was certainly understanding of how useful this could be, but not enough human resources to do so (to intertwine typechecking with expanding macros). In the JS case, it would be interesting what happens but for now the "immediate" goal is to get people to notice sweet.js and stop the insanity of piles and piles of complete-source to complete-source "transpilers" and start doing the damn thing properly.

But type annotations are probably harder to parse than even the rest of the language.

It's probably not too hard for an environment that knows about identifiers etc. The real problems will happen in case of slightly different parsings being done by these things (TS vs Flow), but even in that case, sweet.js could just stick to some agreed lcd.

disnet commented 8 years ago

@elibarzilay knows what's up.

The right thing to do is extend Sweet with support for TS/flow syntax. This actually isn't too hard, certainly not nearly the same difficulty as the rest of the macro system. Just need to add a few cases in the parser and the codegen to handle type annotations.

The obvious concern with going down this path of supporting "downstream" languages in Sweet itself is that handling n languages in our parser (that really should just be macros) is untenable. I think TS/Flow are big and close enough to warrant this embrace and extend approach. Maybe eventually Sweet will get good enough for that other "e" :)

elibarzilay commented 8 years ago

I completely agree. I think that TS/Flow are exceptions since they are a language extension that is largely unrelated to macros. For other transformer libraries, it would be nice to start seeing them convert the code to go through Sweet as a much more logical choice as usual with macros being local transformation rules. When that becomes popular enough (and I might be idealistic here, but I'm convinced it will), then people will finally "discover"[*] that they can do some actual work as macros -- to the point of re-implementing TS/Flow on top of sweet.

([*] And this will definitely take some time. Even in the Scheme world, I remember talking to someone who was very surprised that in Racket we'd do something crazy like invoke GCC as part of the compilation -- in most people's minds, that's way beyond what a sane macro system should be able to do. They're just not used to separate compilation (Racket's phase separation) making things robust enough that anything goes.)

KiaraGrouwstra commented 8 years ago

Regarding TypeScript as "some babel modules + compilation errors" for a second, I'm getting the impression that, in a similar vein as you've mentioned, existing babel modules essentially represent ES-next syntax implementations that would be awesome to have available in Sweet as well.

This leads me to a question, answers likely clear to you guys, but not so much to me as an ignorant but curious user.

if ES proposals / babel models represent new syntax desirable to incorporate into Sweet at one stage or another, do these existing babel modules contain the info/functions required by Sweet?
If so, could they be used/converted to prevent duplication?
Otherwise, what additional logic is required that existing babel module implementations do not expose?

If the additional information required would be trivial, perhaps it could become the norm for the community to provide implementations supporting both babel and sweet for future ES proposals.

dead-claudia commented 8 years ago

I was thinking: would it be a better idea to, instead of adding more and more things to the parser, allow an Acorn-style syntax extension mechanism? That might work better, since you can validate anything you come across. Maybe something like this?

// Just validates.
syntaxdecl async = function (ctx) {
  // ctx.expectCallable(true or undefined) -> include newlines
  return ctx.expectCallable(true) && ctx.is("expr")
}

syntaxdecl inline `::` = function (ctx) {
  return ctx.next("expr") &&
    ctx.next(this) &&
    ctx.next("expr") &&
    ctx.is("expr")
}

disnet commented 8 years ago

Right, so worth taking a moment to explain the philosophy of Sweet (and macro systems in general).

Consider the following languages:

ESyyyy, where yyyy is the current year, is defined by the standard ECMA262
ts, can mostly be thought of as ES2015+types
flow, can mostly be thought of as ES2015+types
babel, can mostly be thought of as ESyyyy+extras where extras are things like jsx, object spread, and various other pre-stage 4 proposals (at which point we get ESyyyy+1) and experiments

Sweet is a parser for yet another language: ESyyyy+macros. The cool thing about the +macros part is that you can define syntax transformers (aka macros) which allow you to accept many more languages than just the strict grammar defined by ESyyyy+macros would imply.

The goal of Sweet is to support the creation of all the languages listed above (and many more) via syntax transformations that you can compose and reason about (acorn’s plugin approach is a non-starter because it is neither composable nor reasonable).

A non-goal of Sweet is to bake in support for every JS-adjacent language into the parser. That’s what macros are for.

Practically, the kinds of syntax transformations Sweet currently supports are not powerful enough to implement all the languages listed above. That said, the plan is to make them powerful enough. Operators (**, :: etc.) can be handled with #517, more complex operator-like forms can be handled with the re-introduction of infix macros, jsx can be handled with readtables, types can be handled with modules #233 + some stuff from Racket’s syntax zoo (the syntactic form of types is straightforward to handle it’s the type checking that requires the extra stuff), object spread can probably be handled by something like Racket’s implicit forms.

Basically, the roadmap is steal everything from Racket.

What we’ve been discussing in this thread is extending Sweet to support the language ESyyyy+macros+types as a temporary kludge because the +macros are not (yet) powerful enough to implement the +types. The only reason we’re considering this is because TypeScript/Flow is a relatively big enough deal for people who might be interested in using macros. It’s a way to bootstrap us into taking over the world (muahahaha).

dead-claudia commented 8 years ago

@disnet Okay. I'm not that invested in that idea of extensible syntax, anyways.

It’s a way to bootstrap us into taking over the world (muahahaha).

:laughing:

KiaraGrouwstra commented 8 years ago

To expand on my earlier comment, if all ES-next proposals would require non-trivial reimplementations as macros, I suppose that means Sweet would essentially be competing with Babel for features added. If so, then adopting Sweet unfortunately for the time being becomes more of a trade-off rather than a straight-forward decision. Obviously, its potential is orders of magnitude bigger, so I definitely hope Sweet could overtake Babel as the go-to way of implementing new features. Until that time, there may be a threshold of momentum though. But then again, I guess for now the bigger step is still that Racket roadmap...

dead-claudia commented 8 years ago

@tycho01 Almost every ES proposal at the syntax level is possible to implement, and many already have with previous versions of Sweet. This includes ES6 classes in their entirety and arrow functions.

Oh, and Babel isn't likely to disappear. Compilers are much better at semantic optimization and even implementation correctness than macro processors in terms of specifications. Good luck implementing generators in pure Sweet. :wink:

vegansk commented 7 years ago

Are there any progress with typescript support?

disnet commented 7 years ago

@vegansk various background tasks have been completed but nothing directly on supporting TS/Flow in sweet.

I'm definitely motivated since everything I write now is in flow (even sweet core!).

My current focus is updating our internal AST to the latest version of Shift (so we can support async/await). Once that has been handled we will be in a good position to support types.

If anyone wants to help out a good place to start is getting TS/Flow support added to Shift. We depend on shift codegen to render our AST so it will need to be extended to handle types.

gabejohnson commented 7 years ago

@disnet are you thinking just matching what Babylon does (maybe more granular TypeAnnotation node types)?

disnet commented 7 years ago

@gabejohnson I haven't looked into it in any depth yet but yeah that would probably be my initial approach.

slotik commented 7 years ago

@disnet I'd like to help with this. I looked at the Swift issue list but couldn't see any feature request related to type annotations. Should I create such a request or is there a better way of adding the support you need?

disnet commented 7 years ago

@slotik no need to open a shift issue. I chatted with them (sorry, forgot to update this thread) and the shift project is uninterested in type annotations in their core AST and that's actually fine for our purposes; we are already extending the shift-ast-spec in a couple of places (syntax declarations and import for syntax) so a few more is not a problem. The only code from shift we are really depending on is shift-codegen so once we get to the point of implementing type annotation support we will probably need to fork it.

As far as what we can do right now I'm sort of blocking anyone helping out with code. I'm in the middle of refactoring sweet-spec and sweet-spec-macro, which will have far reaching consequences in sweet-core.

What would be super helpful though is taking a look at the shift-spec and proposing what nodes we should add/change to support annotations.

slotik commented 7 years ago

@disnet I started to look at shift-spec and I made a couple of notes around the places that seem to need changes:

https://github.com/slotik/shift-spec/commit/980114c03a1f956abd3e0190d89105cd09807ee3

If this looks like it could provide some value, I can try to suggest concrete modifications as well as a new set of nodes to express interface and type statements.

KiaraGrouwstra commented 7 years ago

What would be super helpful though is taking a look at the shift-spec and proposing what nodes we should add/change to support annotations.

For reference, TS AST nodes.

disnet commented 7 years ago

@slotik looks like a great start! Definitely take inspiration from the TS/flow AST nodes. I'll try and finish up my background work on sweet-spec soon.

slotik commented 7 years ago

I think I made quite some progress already, though some loose ends remain:

https://github.com/slotik/shift-spec/commit/bc3e3e7044d6dff1b9639ef6f8e2fe112f469aa0

I'll try to finalize that as soon as possible.

I've modeled most of the missing nodes the way they are modeled in TypeScript. I've omitted everything related to JSDoc and JSX - maybe that is OK?

Anyway, it's a bit hard to tell whether what I've done is actually correct. If anyone dares to look, I'll be happy to adjust as needed as well as try to explain why I did what I did.

disnet commented 7 years ago

@slotik awesome! Feel free to open a PR on sweet-spec with your proposed changes and I can try and do a detailed review. I literally just updated it to ES2017 (sweet-js/sweet-spec#3 and sweet-js/sweet-core#740) so your changes should apply nicely.

KiaraGrouwstra commented 7 years ago

If one were to take the position TS has subsumed all use-cases of JS/Flow, might it become preferable to just add this in the TS compiler rather than maintaining a separate copy of much of the functionality their compiler has already implemented?

I realize that isn't the position taken here, and fragmentation of efforts is sub-optimal as well. I fear I don't have the expertise to judge the question though, so I'm curious as to your take on that @disnet.

Edit: either way, note that parallel to Sweet's is* / from* functions, TS already exposes functions like isNumericLiteral and createNumericLiteral as well, alongside update* variants potentially similar to sweet's syntax transformers, all exposed so others can use them in their own transformers, with stuff on lexical environments in there as well. I'm also seeing a request on support to integrate custom transformers, which would've seemed like a potentially clean way to handle this.

The overlap in functionality seems somewhat significant. Going down the list of the Sweet reference, where Sweet Syntax mirrors a TS Node:

syntax <name> = <init> / syntaxrec <name> = <init>: no equivalent yet
transformer : (TransformerContext) -> List(Syntax): TS's update* functions / visitors sound related?
syntax templates e.g. #${ctx.next().value} + 24: no present equivalent, I guess template strings seem similar but don't magically serialize interpolated nodes
checks like isStringLiteral(s: Syntax): boolean, already exposed by the TS compiler
node factories e.g. fromNumber: already exposed by the compiler as create* functions
unwrap: sounds a bit similar to TS's emitter to serialize nodes back to code
special import/export handling for syntax: might just hook into how these are already handled for types

Edit: oh, TS's exposed program.emit already allows passing custom transformers, but these only cover existing nodes, while Sweet allows adding new nodes.

disnet commented 7 years ago

If one were to take the position TS has subsumed all use-cases of JS/Flow

heh, I definitely do not take that position but am 0% interested in getting into a flame war. TS is great! Flow is great! JS is great! Use whatever makes you happy.

might it become preferable to just add this in the TS compiler rather than maintaining a separate copy of much of the functionality their compiler has already implemented?

The only "functionality" we're talking about adding to sweet here is the ability to transparently pass through type annotations. No type checking or other fanciness. This allows sweet to focus on doing it's one thing (expanding macros) and let other systems consume the output of sweet and do what they do best.

In terms of feasibility of integrating sweet with other compilers like TS, this is...non-trivial. This has already been pointed out in this thread https://github.com/sweet-js/sweet-core/issues/482#issuecomment-205180418 so I won't bother expanding on it.

KiaraGrouwstra commented 7 years ago

I see. I don't know much about the intricacies of hygiene, but it appears that TS handles it in their createTempVariable function, name passed back to a callback currently used in a couple 'transpile to ES*' transformers for hoisting purposes. I don't fully know the Sweet side of the story, but with that, TS's stance on hygiene may be more nuanced than "don't care" as @elibarzilay phrased it.

But yeah, even if they allow passing custom transformers, it does appear that extensibility of e.g. the parser with new nodes would remain an issue there...

I wonder if external handling of macros might come at a price of its own though. Some IDEs facilitate the user with type checks, which I imagine would break down if they weren't able to handle the macro part as well. The approach suggested here so far leaves me wonder how that might work in practice.

gabejohnson commented 7 years ago

The only "functionality" we're talking about adding to sweet here is the ability to transparently pass through type annotations.

We're talking about the "ability" to transparently pass through the annotations though, not the necessity. Right? These are still going to be reader tokens -> AST nodes.

If that's the case, we're really talking about hardcoding something that could be implemented as a language by using a custom reader and some macros (type, interface, etc.).

I'm not suggesting we don't support TS natively, but it's something to think about. This could be a good test case for exposed reader macros.

disnet commented 7 years ago

it appears that TS handles it in their createTempVariable function

Nope :)

That's not the hygiene @elibarzilay was talking about; it's just gensym as the schemers would say. Hygiene refers to the process of tracking bindings and references during macro expansion. The machinery to do this tracking is surprisingly subtle and impacts the design of a compiler in far reaching ways. It's not exactly that TS compiler folks "don't care" so much as the cost of supporting macros (in the tradition of Racket) is probably not worth it from their perspective.

I could be wrong of course and if someone wants to add a Racket-style macro expansion system to TS go for it! I'm not interested in tilting at that windmill though.

I wonder if external handling of macros might come at a price of its own though. Some IDEs facilitate the user with type checks, which I imagine would break down if they weren't able to handle the macro part as well.

Regardless of where the macro expander is (internal or external to the type system), the macros must first be expanded. This seems to me fairly straightforward to handle with a plugin that first runs sweet to expand the macros and then run TS/Flow to type check. Sourcemaps can stitch everything together.

KiaraGrouwstra commented 7 years ago

Thanks for elaborating.

disnet commented 7 years ago

These are still going to be reader tokens -> AST nodes.

Of course. The real issue is not reader macros, correct me if I'm wrong but I don't think we need readtables at all to handle type annotations, there's nothing lexically different about them. The real issue is the resulting AST and codegen which is why we need to hard code types for now. We want the output of sweet to be something the existing type systems can consume.

Eventually we could implement type checking itself entirely in sweet as just a collection of macros that communicate type information during macro expansion just like typed racket.

KiaraGrouwstra commented 7 years ago

Eventually we could implement type checking itself entirely in sweet as just a collection of macros that communicate type information during macro expansion just like typed racket.

Well, that'd be quite the feat as well; their checker.ts currently weighs in at a whopping 1.36 MB of code. I've been trying to figure it out, but there's quite a bit in there including this backward 'contextual' inference that complicates things a bit further, the details of which still elude me. And despite its size, this checker hasn't quite nailed all the details yet either.

gabejohnson commented 7 years ago

I don't think we need readtables at all to handle type annotations

You're right. I don't think that TS/Flow do anything different with /.

The real issue is the resulting AST and codegen which is why we need to hard code types for now.

Disregard those comments, I was having a moment.

Adding the type nodes would actually be really useful for creating alternative annotation syntax (HM perhaps).

slotik commented 7 years ago

Here's the pull request for augmenting sweet-spec with TS-style annotations.

elibarzilay commented 7 years ago

@tycho01, the gensym functionality that you've referred to is important to implementing hygiene, but it is not sufficient by itself. There are piles and piles of Common Lisp vs Scheme flamewars that focus on exactly this point (CL has gensym, Schemers point out its shortcomings when compared to hygiene).

As for the TS functionality that you're talking about, I don't know much about it (just saw it mentioned in the thread you posted to, since you've mentioned "hygiene" which is one of my trigger keywords), but it sounds like a kind of a simple compiler plugin thing. That too is something that is needed to implement macros, but it's the lowlevel thing that makes it possible. Probably the most obvious way to see the difference is to consider how uses of the TS thing would be implemented in their own files and possibly even described in some special way in meta-information files (eg, invoke the compiler with some magic that loads in a specified plugin) -- then compare this to a proper macro system where not only you don't need any special meta tweaking, but more than that you can mix in plain code with macro code in the same source code or even the same expression (with a local syntax definition).

In my experience, people who are talking about "preprocessing" are almost always completely unaware of the difference when they assume that macros are the same thing. I expect some people on the TS team know the difference, but in the same way I expect them to not be interetsed in implementing it if only because their target audience (your average JS developer who knows some statically typed language) doesn't really care.

I very much hope for a future where this is different, but it'll take a bunch of time for people to get educated, and IMO sweet.js is the most promising effort in getting the point across. (Sidenote, I recently looked again at Scala macros, and again I was surprised at how even in that neighborhood there is a huge gap between what you can get when compared with a "proper" macro support in the style of Racket.)

Wizek commented 5 years ago

What's the current situation with TypeScript support? If I just want to add some simple macros to some TS files, is it currently possible to do Sweet -> TSc -> JS or TSc -> Sweet -> JS? Or would the two compilers get confused by each other's syntax no matter which one gets to compile the file first?

letharion commented 5 years ago

@Wizek I think it's safe to say Sweet is mostly abandoned. No commits at all for 2 years, no recent PRs, no recent comments on issues from a developer. Thus, I think no TS support is forthcoming.

Wizek commented 5 years ago

Ah, thanks for the heads up @letharion! In that case I am quite glad I didn't sink much time into learning Sweet.js! I'm just now looking at https://github.com/kentcdodds/babel-plugin-macros; is that the closest alternative or are there others?

vendethiel commented 5 years ago

No, that module is very very simple and doesn't provide anything like custom readtables, custom operators, etc. It just avoids you having to write boilerplate to make a babel compiler plugin.

sweet-js / sweet-core

Support TypeScript/Flow type annotations #482