Committee feedback around grammar complexity is getting ignored

[theScottyJam]

Feedback from TC39

The following is a list of quotes from various delegates, asking for this proposal's problem space to "be open to very simple syntaxes" and "open to exploring syntaxes which are very different from TypeScript". These quotes are pulled from the March 29, 2022 and March 31, 2022 meetings when this proposal was seeking to obtain stage 1.

A selection of quotes related to reducing the grammar size

From Waldemar Horwat > What's being proposed is not a small amount of syntax. It's maybe half or more of TypeScript. If the goal of this were to add a new way of having types of comments, I would expect maybe two or three grammar productions, not pages and pages of them. > I would not want to exclude [syntax] which [is] much simpler than what TypeScript supports [from being considered]. > I would like this space to be open to very simple syntaxes. From K.G. > if the goal is just to have some syntax where you can write types, there are simpler options, and I would like the focus to be on those, so that we reserved a much smaller amount of syntax space. > I would just like to ensure that we are saying that we are open to exploring syntaxes which are very different from TypeScript specifically, because TypeScript has so much syntax. From M.M. > I would certainly very skeptical of any large grammatical addition to the language. From M.F. > it seems like it is unlikely that the committee will end up opening some huge syntax space basically as we saw in the original proposal [...] It's more likely that we see a narrow space, though an infinite one, added for these kinds of forks to make use of. > Now my question to the TS team is, if such a narrow space was added, would TypeScript be willing to actually move toward using that syntax space that gets reserved, or is really the only option for JavaScript to become TypeScript?

I'll also want to discuss a bit about how coupled this proposal is to TypeScript - I'm mostly doing this because "TypeScript has so much syntax", and decoupling this proposal further from TypeScript is a necessary step if we want to reduce the amount of grammar this proposal requires - a step that many delegates were asking for, and IMO hasn't really been taken yet (aside from removing TypeScript's name from various places in the README and what-not).

A selection of quotes related to decoupling this proposal from TypeScript

From George, (who develops the Flow language) > As you can see in the grammar, which specifically is all of TypeScript and encodes typescript specific features and codes. None of flow specific features. > In reference to any flow specific features, it suggests that Flow can simply change its syntax. > I would say that, you know, to go to Stage 1, we should make it clear that no special treatment will be given to any existing type Checker. From Chris de Almeida > [We have] some reservations about the proposal and the coupling to TypeScript

How is the proposal addressing this feedback?

Today a community meeting was held where they talked about some of the past feedback they received from prior TC39 committee meetings and what they were doing to address it. The main focus seemed to be feedback related to the grammar itself and some of the inconsistencies it created. They shared this tentative grammar document and explained how this updated grammar was tweaked to address some of these previous parsing issues, Which, that's great that we're getting the small grammar issues fixed.

But what about the larger concerns?

From their slide, they had this quote:

We continue to see supporting compatibility with types used in current ecosystem, e.g., TypeScript to be major goal

This is precisely what the TC39 delegates were saying in that list of quotes shouldn't be a major goal. They were specifically asking for this proposal to be willing to explore options where compatibility with existing type systems wasn't very strong.

The conversation later moved to a discussion around what was being done to satisfy the concerns of other typed supersets (flow, clojurescript, etc), and it seems like the answer to that is, currently, very little. They did prepare this spreadsheet comparing the syntax of different typed supersets, which is a step, but their currently proposed grammar document they had shared doesn't seem to reflect anything from this spreadsheet.

During the meeting, George (the Flow developer) pointed this fact out - that the grammar still has many productions specific to TypeScript and no flow-specific grammar like "opaque" types. The response was that they could consider adding it, but maybe George's team could also consider changing Flow's syntax to move this "opaque" keyword so it would fall into the existing flexible token soup. To back this up, they mentioned that TypeScript was also changing some of their syntax (they cited the change to generic parameter syntax to be a turbo-fish operator) - but the thing is, TypeScript is required to make that change to avoid "breaking the web". The only alterations I'm aware of that TypeScript is currently considering are alterations like these for web compatibility reasons or to drop legacy features they don't care much for anyways, so it felt very odd to hear Flow get asked to make this sort of sacrifice to make their grammar fit into the token soups, when the tentative grammar doesn't have TypeScript making any sacrifices like this.

What can be done instead?

We can have TypeScript make the same kinds of sacrifice that it was asking Flow to make. We can let TypeScript move some of its syntax around into token soups in order to minimize the number of grammar changes being made by this proposal.

This can take on many forms - many of which have been discussed by bystanders here in the GitHub issues. In the earlier TC39 meetings I was quoting from, K.G. gave an off-the-cuff example of what could be.

So for example : is already reserved in several positions, just [...] make it [also] a comment in some positions. And then you could write, know, :interface as your declaration or whatever. This wouldn't make a typescript work, but it would allow you to write type declarations in your code with very little syntactic overhead.

i.e. instead of writing

interface {
  x: number
  y: number
}

K.G. seems to be suggesting that the grammar could be written so that you instead write this:

:interface {
  x: number
  y: number
}

Is that backwards compatible with TypeScript? No. Does it support the "major goal" of being "compatible with types used in current ecosystem"? No. But it does provide a big leap forwards in making the grammar both simpler and more flexible.

Conclusion

Please, please, please consider the delegate feedback. I would really like to see the problem space of "unfork JavaScript" being explore more broadly than what's currently happening. There was lots of pushback around this proposal when it was initially presented, asking the champions to be open to other options that they weren't originally considering, and it's disheartening to see that the up-to-date rough-draft grammar is just a more refined version of the stage 0 proposal.

And yes, I get that "explore other options" doesn't mean "you're required to go down a different path", but if you choose to stay on the same course, I hope there's a very strong and well-thought-out argument explaining why this is the case, that can address all of these concerns the delegates were bringing up. Something a little stronger than "We continue to see supporting compatibility with types used in current ecosystem to be the major goal", because that's not supposed to be the major goal. From my reading, stage 1 was conditioned on that not being the major goal.

(Take everything being said in this post with a the attitude of me being worried and a bit disappointed, but still hopeful that these issues can be properly addressed. This isn't intended to be accusatory or anything.)

[egasimus]

What is a "token soup"? Do I understand this correctly - an escape hatch from the language's grammar? Like comments but meant to be handled by different tools? And then, to slowly fossilizing into the language? Who introduced this concept and should it ride on the coattails of a type annotations proposal?

[trusktr]

188 has lots of examples of areas where arbitrary characters can be placed, with only certain markers terminating the "comment space" and transitioning back into code space. I think that's what token soup means if I understand correctly. And yeah, there's no specification of what goes in the comment space, except for the characters that terminate the space. Different type systems can have radically different type syntax within those spaces.

[theScottyJam]

What is a "token soup"?

The current proposal has "token soups" whenever you use a pair of brackets in the comment space. For example

const x: (I can | put & almost <anything> in here!! and JavaScript will just ignore it ^&*#$) = 2;

This is supposed to be the flexible space for type systems to add their existing syntax, or new future syntax, without requiring TC39 proposals to explicitly update their grammar for each little syntax change.

They've had this feature since the proposal was birthed, but this term wasn't coined until later - I believe it came from this presentation and then it sort-of stuck. It was brought up in relation to some concerns around it, related to how you might accidentally "comment" out more of your code than you intended if you aren't careful - it's an interesting read.

[azder]

@theScottyJam @egasimus funny how I got an idea about re-using reserved keywords and before I'm done with some initial musings, I got the "suggestion" or "counter argument" from someone else.

Will this solve the token soup? Just allow any interface language to co-exist side by side?

https://github.com/tc39/proposal-type-annotations/issues/176#issuecomment-1732277770

[egasimus]

@trusktr @theScottyJam

Thanks! So, like comments but meant to be parsed? Very interesting approach! This thing could be used to bolt unlimited extensibility onto an existing programming language.

Hence, "portal to hell". It feels like the sort of thing that will have tons of unexpected ramifications. It deserves to at least be discussed in a proposal of its own.

It's a possible step towards adding metaprograming to JS. That's a risky area, as among the current mainstream of practitioners there doesn't seem to be a big overlap between those who are comfortable with macros and those who are comfortable with category theory. Even if the approaches may be made complementary.

Even if types could've been implemented in terms of macros, we do have the reality that they're currently being implemented in terms of compilers - and native macros would work best in a workflow where a compile step isn't the default.

My opinion is that it might make sense to keep an extensive development such as adding extensibility/metaprogramming to the language, orthogonal to an intensive development such as static typing which is meant to rein in the existing complexity, not open up exciting new worlds to make mistakes in.

(Could go with a less cozy name, too. Ima call "token soup"-type constructs excessions because they let you exceed and exit from what's specified by the language. As in, semantic comments are the most common kind of excession; and Rust doc comments are an example of how to have native comment-based documentation without excessions.)

@azder @msadehque

Good thinking on taking stock of existing reserved keywords! I love the idea. Also linking #196 here because it can be difficult to keep track.

[azder]

My opinion is that it might make sense to keep an extensive development such as adding extensibility/metaprogramming to the language, orthogonal to an intensive development such as static typing which is meant to rein in the existing complexity, not open up exciting new worlds to make mistakes in.

☝️ Fully agree

On the name, everyone has been basically trying to not say DSL or ability to plug in a DSL, but that's basically it. Just like RegExp is a DSL for matching, this one proposal might have started as an intention to provide a way to plug in typecheck DSL, but as you said, it opens the possibility to anything and everything be added in a place where maybe it shouldn't.

Not that hard to imagine someone plugging in SQL after your function parameter name. Like a half page long one