CS2 Discussion: Project: Project Goals

[kirly-af]

We are struggling to know what we want for this project. This is discussed in different issues and I thought we should make a dedicated issue.

The way I see the project is:

1. we integrate the core features/breaking changes defined in #8 (those marked Tier 1, and ideally Tier 1A), enabling us to produce ES6+ output (until that stage, this could be simple PR to the original CS repo as @rattrayalex said).
2. once we have this sane basis, to me we should make this project be the CoffeeScript Babel: plugable (#4), with new features implemented as plugins.

Then we need to think how to make the new features be introduced in the language. We have no tc39 to decide what will be standard and what will not.

I guess the ESx proposals should be core plugins that we would integrate in the compiler once standardized (of course I'm talking about those that would need modifications in the CS compiler to work).

The question is: what should we do with independent feature ideas ? The list of feature ideas is growing (see #16), so we definitely need to define a process to review those ideas.

I think we could adopt the tc39 process. For that, we would need a fixed list of legitimate people to discuss the proposals (+ possibly some people from the community ?).

Any thoughts ?

[rattrayalex]

Thanks for opening this @kirly-af . Well framed and well said. I think the timing is right as well.

I personally certainly agree with aim 1. I hope to put together a PR in the next couple days that summarizes that and integrates it into the Readme.

As for 2, I like the babel-style, pluggable-coffeescript idea for developing and testing features, but I think we need more thought and discussion around that. I'd feel comfortable leaning on that as a tentative path forward, at any rate, if there are no objections.

I'd really like maintainers of the official coffeescript repo to be the ones guiding a tc39 process, and I agree that such a process would make a lot of sense. Perhaps a few members of this community could be promoted by @jashkenas to that role, but that's up to him. I personally would not volunteer. In any case, I'm sure such a request (both adopting the tc39 process, and installing members of this community onto a deciding committee) would be much more warmly and credibly accepted once we have sent successful PR's to jashkenas/coffeescript filling in the "Tier 1" features like import/export, classes, etc.

Thoughts?

[carlsmith]

At this stage, I think preventing CoffeeScript from becoming obsolete is the only top priority. We can work on extending the language once we've assured it wont become obsolete, which is a serious concern.

I'm happy to have these conversations about language design, to try and create a resource for the future of CoffeeScript and similar languages, but the focus of any work should be an ES6 option for the official compiler. Building a whole new version of the language with novel features is a huge project that could easily be rejected by the community.

CoffeeScript developers are adding ES6 features and we can help with that too. We should just focus on doing a fork of the current code generator, make a breaking change to make classes work with ES6 properly, and then incrementally modernise the output to catch up with and then follow the latest standards.

We still don't have anyone who wants to get that started though. Someone will have to get that process rolling, take ownership for the project long term, and lead development, else it's never going to take off or it won't accepted into the official codebase.

[MemoryChips]

I agree targeting an ES6 should be the goal.

In addition, I think coffeescript was a great success because it had effectively a BDF. Without a dictator, you will end up with too many distractions. All of which are usually very good but you have to choose or nothing will get accomplished. Rob

[objectkit]

Hi all, its really good to see this thread here! Personally, I've found it difficult to understand and keep to the flow of this discussion so far as its so multidirectional, partly due to my own noise and difficulty interpreting the flow of discussions, so good call @kirly-af! I'd like to see some kind of formal process or agreed set of commitments to establish a sane basis of progress soon enough.

For whatever its worth, I'm in total agreement with aims 1, 2 and 3, but perhaps not in that order of execution.

In an ideal world, is the following high level path plausible? Could we collectively agree to

• research and pick an established third party open source general purpose parser/lexer/compiler
• define the official CoffeeScript specification as-is with a formal grammar compatible with the compiler
• choose an output language by picking an interoperable transpiler mechanism compatible with the compiler ('make it pluggable')
• decouple new CS language specifications and the development of them from ES as much as is realistically possible

Ideally, if the above is possible, when a new CS language specification is produced, and a compiler or family of compilers in place, we will still end up with choice of input language (CS1, 2, 3, 4...), and choice of output language (ES4, 5, 6, ASM...).

It would make sense in that context to most definitely pick ES6 as one of possibly many destination languages in a loosely coupled way. Who knows, by the time we agree on CS 2 (!!), ES7 may be a very viable destination language, but ES6/5/4 would still be there if desired.

To my eye, a path like that could help us reach aims 1, 2 and 3.

Apologies if I've annoyed anyone by saying that. My intention is purely for love of CS and I am really quite passionate about wanting to see it continue to grow as a language in its own right, and not just a DSL, even if not in a direction I personally prefer.

Namaste!

[carlsmith]

By ES6 support, we mean ES7 once that is more widely supported, and so on. It's just that the current compiler was designed in an era where we wrote cross-browser code by hand, and we now want to generate the latest JS, and leave polyfills to other tools.

Building a whole new extensible compiler from scratch is a huge project, that will probably never become a drop in replacement for the current compiler, so will remain unlikely to be adopted.

If you want the lowest hanging fruit and a high probability of adoption, reusing the official frontend, ensures continuity and compatibility with the official language without a big fork dividing the already diminished community.

[objectkit]

@carlsmith I hear you and understand your perspective- leaving polyfills to other tools, and outputting to ES6 for that purpose makes sense. Never doubted that. Fully aware. As I am technically a dinosaur with firsthand experience of NetScapes LiveScript as a young teenager, I'm well used to its difficulties and hurdles. I'm not suggesting building a whole new compiler from scratch, just to research whats out there as an alternative to CoffeeScripts current compiler and to consider using a new formal grammar representation as a result. There is a difference between a language specification and its compiler as we know. At present, they are tightly bound in the world of CS, which I perceive to be one of the hurdles for its continuation. Unfortunately I suspect that building on the current compiler will only continuate that hurdle.

If there was adoption of a new compiler from CS official after research and proof of concept, I imagine the uptake would be ok as it would permit the language to evolve, rather than diversify into dialects which is the current hazard.

So for the record, I hope the CS community that still exists is there for the language and wouldn't balk at a newly appropriated compiler (not built from scratch) that could understand a formal grammar representation of CoffeeScript 1, from which its AST could be transpiled, even solely to ES6 for sake of making it available as an input to Babel today.

I guess thats the fork in the road right there.

[carlsmith]

If a new compiler was more realistic, it'd be different, but having seen Redux fail to pick up users, when its goal was to replace CoffeeScript with a compiler that targets the estree AST format, it seems better to not aim so high this time if we can avoid it. I'm not against a new parser project, but want to try and push for the most doable solution to the main issue. It's not that you don't make a lot of good points, but it seems best to keep things as simple as possible for now.

[objectkit]

Hey... I think we have the roughly the same objective - attempting to lend our time to the preservation of the language and its evolution in an organised linear fashion for the benefit of many - I'm just not convinced that repurposing the current CS compiler will have long term benefit or that it is necessarily a de facto simpler path forward. No doubts it will get you there eventually, but I suspect it will incur its own difficulties as there isn't enough decoupling between the language specification and the present compiler. To cut to the chase, I believe attempting to

(a) define a new informal language specification, CS6, to suit (b) the constraints of a preexisting DSL specific compiler derived by forking CS, while (c) committing to one output language, ES6,

in one go... as a one off end in itself, yes, I think you'll get there, but the cost is high and the constraints uncertain.

I don't see simplicity in that. And I haven't done my homework sufficiently either! That is, there isn't a decided upon research proposal to investigate the feasibility of appropriating a general purpose compiler, so my hands are in the air, I only (strongly) suspect that as a plausible direction with long term benefits. But I'm thinking purely in terms of the language here from a research perspective, not project user acquisition. Perhaps that again is another fork in the road. Best of luck Carl. I think you'll get there.

edit: I'm being sincere, no undertones - the route you are planning is probably achievable - pax :)

[zeekay]

Rather than build a new compiler I'd focus on creating a parser that can be swapped in for Babel's parser, Babylon. You could use Redux or decaffeinate's parser as a starting point and update it to output the Babel AST (which is based on the ESTree spec).

Focusing on just the parser part of the compiler stack makes a lot of sense to me -- less to build and maintain. As a bonus it would make it possible for CoffeeScript to easily interoperate with the existing Babel ecosystem and ensure adoption of new features is rapid.

[carlsmith]

Doing a new parser is interesting, but my concern would be that it'll not be compatible with the official compiler, so you'll have another dialect that can't be used as a drop in replacement for parsing existing code bases. Will anyone use it? Maybe. If you do a good job of it, then perhaps, but building an ES6 code generator for the official compiler that is selectable with a flag would be much more likely to solve the core problem.

The advantages of Babel are not lost. We can write a transformer that walks an official CoffeeScript AST and generates one for Babel or whatever we need. We're doing our own code generator, but can reuse any of the existing parsers to generate our AST. Using the official one will be least likely to cause division, as people will only have to update their classes to move a module to the new compiler.

[zeekay]

I do agree it's a good idea to re-use an existing compiler/parser, but I think we should target the Babel AST. Code generation is best handled by Babel. This will enable more granular control over the code generated (Babel does this very well), faster compilation, easier adoption of new features, etc.

[rattrayalex]

CoffeeScript developers are adding ES6 features and we can help with that too. We should just focus on doing a fork of the current code generator, make a breaking change to make classes work with ES6 properly, and then ... catch up with ... the latest standards.

Yes.

This issue was opened asking for project direction. Here's the project direction.

First, we're going to focus – as individuals, or as a team – on getting ES6 modules (import/export) and classes into jashkenas/coffeescript.

@GeoffreyBooth and @JimPanic have been collaborating on the former here and have welcomed others to join. I'm not aware of a leading effort for classes; please post one on the relevant thread.

Once that's complete, we can consider doing the same with async/await and const, if @jashkenas is interested in the suggestions there (-> for async, <- for await and := for const), and any other similar features.

Other than that, I think anything interesting is going to take strong, opinionated leadership and probably breaking changes. I'm not willing to provide that myself, nor am I comfortable appointing someone.

As such, I propose the following process:

1. Anyone interested in leading a new "CSNext" effort submit a proposal as a PR to this repo. This will be a markdown file in a proposals directory.
2. I'll evaluate it for clarity (not whether I think it's a good idea), merge it, and create a new repo for them under the coffeescript6 org. I'll also link to each repo from the main README here.
3. We can continue to use this repo and the csnext gitter for discussions about the future of coffeescript.

An effort can take the form of a completely new language inspired by CoffeeScript, a fork of jashkenas/coffeescript, a fork of coffeescriptredux, a from-scratch Babylon/Babel CoffeeScript parser, or whatever.

There are a lot of great ideas floating around this repo and I think it's time we put a few to the test.

I'll be submitting a PR shortly that adds this to the repo and creates a proposals directory, and will update this comment with a link when that's done. Until then, I'd love thoughts and feedback!

[carlsmith]

As you know though, I don't think that's the best plan. Adding classes to the original compiler will not work the same way as the other features, because classes break BC. If we just incrementally add features to the original compiler, then classes will end up with their own compilation flag. I think it's better to do an ES6 code generator, and have a flag for that instead. Otherwise, we're going to keep outputting shimmed ES3 for the bulk of our code, even though that code depends on ES6.

If anyone is planning to add support for ES6 classes to the current code generator, it's no extra effort to fork the current code generator, fix classes and use the forked generator as a drop in replacement for the current one. Then, over time, we can all incrementally modernise the new generator. With the current plan, there will not be a new generator to contribute to.

I'm not willing to take long-term responsibility for the project, but still want to encourage whoever does eventually look into fixing classes to not overlook a totally costless, golden opportunity.

[rattrayalex]

I think it's better to do an ES6 code generator, and have a flag for that instead

Go ahead and propose it 😉

Thoughts on the details about how classes should be implemented (eg; whether they compile to ES6 classes, or something else, whether they use a command line flag or a major version bump, etc) should be discussed on the relevant thread – or, better yet, on a PR.

We've been talking a lot. It's time to code.

[carlsmith]

Go ahead and propose it 😉

I have proposed it, many times.

Thoughts on the details about how classes should be implemented (eg; whether they compile to ES6 classes, or something else, whether they use a command line flag or a major version bump, etc) should be discussed on the relevant thread – or, better yet, on a PR.

This is the relevant thread. I'm trying to define the project goals. The point I made wasn't addressed, frustratingly, but it was still made, and this is the relevant thread for that point.

This was meant to be a discussion, not just one person declaring "This issue was opened asking for project direction. Here's the project direction...".

We've been talking a lot. It's time to code.

This is a repo for discussion. That's why we're talking a lot. And the best time to question a bad decision is before you implement it.

[GeoffreyBooth]

I’m tackling modules in the original compiler because it can be implemented without breaking backward compatibility, and in my opinion it’s the biggest impediment to CoffeeScript working with modern frameworks and build tools. ES2015 classes would probably be the next most important feature to tackle, but like @carlsmith mentions, implementing support for those would require either a breaking change or a compilation flag. If we think that this CSNext compiler might take a long time to create, I would be inclined to try to bring in classes into the current compiler and add the compilation flag, as much as I dislike that approach; and then CSNext or whatever it ends up being called would be the next generation of CoffeeScript.

But what it boils down to is what are we doing this for? The overall ultimate goal, I think, is to revive CoffeeScript’s marketshare and mindshare, and keep it relevant for the future. To do so involves solving these problems:

1. ES2015 features that modern frameworks require, like modules and classes, must be supported in CoffeeScript ASAP. We can’t expect developers to continue using CoffeeScript if they must choose between CoffeeScript and whatever hot new framework they want to use for their project.
2. CoffeeScript should support as many features of ES2015+ as possible. There is very little that ES2015 offers that CoffeeScript lacks; but whatever can be supported in a reasonable way, like const, should be supported. We don’t want developers to feel like choosing CoffeeScript means they’re giving up features they had in ES2015.
3. CoffeeScript should output as much ES2015+ syntax as possible. This is much more nebulous, as the upside of this is hard to pin down. It should make debugging easier, for example if a CoffeeScript fat arrow shows up in DevTools as an ES2015 fat arrow; though it won’t, anytime soon, as long as Babel is the final step in the chain (unless Babel is configured to let as much ES2015 through as Chrome or whatever you’re targeting supports). The benefit of CoffeeScript outputting the latest standard JavaScript is that as browsers catch up, and the evergreen browsers support the full ES2015 standard and projects no longer need to worry about older browsers, debugging really will become easier as CoffeeScript’s output more closely matches its input. But there’s much less urgency in solving this problem.

If modules and classes are the only things preventing CoffeeScript from working with most of the most popular current frameworks, then we can fix those two in the current compiler while this CSNext project tackles the bigger, less-urgent problems of supporting new ES201x features and outputting ES201x code.

I would also love to hear from anyone involved in the CoffeeScript Redux project about how far along that project is, and how much effort would be involved in using it as the starting point for CSNext. For example, this pull request. @michaelficarra, do you care to comment?

[kirly-af]

I would also love to hear from anyone involved in the CoffeeScript Redux project about how far along that project is, and how much effort would be involved in using it as the starting point for CSNext. For example, this pull request. @michaelficarra, do you care to comment?

+1

That PR looks amazing. Looking forward to play with it.

[jashkenas]

Very exciting and heady stuff. I'm looking forward to seeing some PRs get merged — perhaps into an "es6" branch, on jashkenas/coffeescript.

That said, I'm commenting into this box from a smartphone in the woods. If you want me to answer a question or respond to anything, don't @ mention me because I probably won't see it. Shoot me an email and I'll be happy to take a look.

[DomVinyard]

into an "es6" branch, on jashkenas/coffeescript.

As per the other thread, esNext branch would be my preference. Let's future proof.

[carlsmith]

When ES6 classes are looked at, I really think you should fork the current codegen, fix classes, and then add the new codegen alongside the old one, then have the compiler flag determine which codegen to use. At first, we'd be shipping two almost identical codegens, but the benefit is longer term.

If someone selects the new code generator, we know they are opting into ES6 dependence, and opting out of backwards compatability. So, we can all start incrementally modernising the output from the new codegen to be pure ES6. We could also make nonBC changes, but they should be minimised to help with migration.

This would mean shipping two almost identical code generators at first, but is otherwise no more costly than just fixing classes. It gives us a place to develop an ES6 codegen that will always work, even if it generates very little ES6 at first. It also reuses the same parser, assuring consistency.

[carlsmith]

Sorry to keep pressing that point, but it would be gutting to see the opportunity lost. It's something we can all contribute to as well. I'm not alone in being willing to help, but not willing to take ownership of a feature, so I expect it'd be a popular thing to contribute to. Please reconsider this.

[JimPanic]

@carlsmith That's actually a good idea and what I had in mind as well.

[JimPanic]

We have to have a path that allows for smooth transition, though. I'm afraid otherwise this endeavour ends like redux. Unfortunately I don't have the time to look at this in detail right now and would rather see import/export to be functional first as this needs some more digging and a lot of refactoring beforehand.

[carlsmith]

Thank you Jim. Cheers mate.

[carlsmith]

Huge thanks for all the work you've put in too Jim. It's a big deal; the future of the language depends on these features. We don't need all of ES6, but modules are killer. If CoffeeScript survives ES6, it'll be you and a couple of others that saved it.

[rattrayalex]

@carlsmith

I have proposed it, many times.

Yes, I noticed 😉

I also noticed that you've struggled to explain your vision in a cohesive way that's rallied the community. Not your fault; its tough to do so when conversations are flung across multiple issue threads and a chatroom.

That's why I figured it might make sense to coalesce thoughts on "what next" in discrete, concrete, complete proposals. Put the big idea and core implementation details in one place, where they can be both updated and discussed.

And, as @GeoffreyBooth sagely asked,

But what it boils down to is what are we doing this for?

We haven't reached consensus on exactly what we're trying to do. How big is the scope? Are we trying to adopt all ES6 features, or only some? Are we fixing warts and adding new features? Are we improving the coffeescript codebase?

These are two many questions, and too complex and interrelated, to answer with consensus. Especially without a formal group with formal membership and formal protocols, which I don't think we're ready for yet.

I know many of us have answers to all those questions in our heads. @carlsmith clearly you do, for example. But we all have different answers, and sussing out the differences and nuances just isn't going to work without a more cohesive proposal, complete with guiding principles and implementation ideas (eg; redux-fork vs rewrite).

So what do you guys say? Are we ready to start outlining specifics of what we're going to do, how we're going to do it, and who's going to lead it?

[rattrayalex]

Separately, apologies if I was a brusque in my declaration of Here's the project direction.. I like that most members of this community have been open-minded and non-hierarchical so far and I'll try to keep things closer to that in the future.

[DomVinyard]

@rattrayalex

So what do you guys say? Are we ready to start outlining specifics of what we're going to do, how we're going to do it, and who's going to lead it?

who's going to lead it is the biggest one. To my knowledge, nobody has volunteered.

[carlsmith]

Hey Alex, don't apologise. I was frustrated, and threw my toys out the pram mate. It's me that owes you an apology really. Sorry dude. As you said, it's been a bit chaotic, but it's been well worth it too, and has made me feel much more optimistic about the future of CoffeeScript.

[GeoffreyBooth]

Okay @rattrayalex, I’ve taken a stab at a proposal. I don’t have answers for everything, but hopefully this can serve as a starting point for discussion.

[auvipy]

have you guys have a look on this https://github.com/michaelficarra/CoffeeScriptRedux/pull/344?

[mrmowgli]

Is there an overview somewhere of the status of CoffeeScriptRedux?

It looks like the original jashkenas parser code seems to be workable, from what I've seen from @GeoffreyBooth and @JimPanic, and if the original parser can be understood or in some way better documented, I would lean towards that.

However if CoffeeScriptRedux has cleaner code, then it really comes down to how much would need to go into just making it work, and figuring out where it is still lacking.

[GeoffreyBooth]

@mrmowgli There’s this roadmap, circa 2014, which looks pretty intimidating. Based on that I’d think that Redux has a long way to go before it’s production-ready. Yet the main README says it’s “complete enough to use for nearly every project.” Maybe the roadmap is out of date? @michaelficarra?

My experience with the original parser makes me eager to use something else 😄 but not if it means finishing a project that was abandoned for a reason.

[auvipy]

the coffescript team supported the work of new parser but later they disagree about targeting es6, thats why ficarra stopped working. but there is a pr for targetiing es6.

[rattrayalex]

Closing as @GeoffreyBooth has clarified this for now, and this thread has become quite long.

I imagine we'll want to refine the project goals periodically going forward, and suggest that we do so in new issues/PR's.

[rattrayalex]

Happy to re-open if others feels we should. Give a thumbs-up to this comment if you would like to see that happen.

[coffeescriptbot]

Migrated to jashkenas/coffeescript#4921