Support CoffeeScript on the new `deno` JavaScript engine

[shreeve]

This is a feature request to see what is needed to support running CoffeeScript on the new deno js engine (see https://deno.land/).

The original author of node is Ryan Dahl (@ry). After a decade of experience, there are several things that he would do over if he could do it all over again de novo (thus, the name deno). There is an excellent video where he describes the rationale to start from scratch and build a modern, secure JavaScript runtime on V8. The deno engine (analogous to what node is), is a single binary executable written in rust that embeds the V8 engine and supports ES6 modules natively. Package management is also greatly simplified.

The deno engine has native support for additional languages such as TypeScript. It does this by automatically transpiling TypeScript code within an isolate and then feeding the transpiled js to the deno engine. Normal JavaScript is fed directly to the V8 engine. While deno isn't done yet, it would be great to be able to bring the beauty and succinctness of CoffeeScript to deno and this issue is an initial attempt to see what is needed to do so.

What I am requesting is to see if we can add support for CoffeeScript to be supported as a first-class isolate like TypeScript.

@GeoffreyBooth or @jashkenas, is this an ok place to raise the question?

[GeoffreyBooth]

I think this is a fine issue to hold discussion about making this happen, sure. I don’t see why we wouldn’t want CoffeeScript supported in deno if that were possible.

CoffeeScript is currently a CommonJS NPM package, so that version wouldn’t be runnable in deno; but the browser version would. I guess the question is, does deno support third-party loaders equivalent to its internal TypeScript loader? If so, then you just package up the browser compiler inside whatever wrapper deno needs for a loader. If not, you’ll need to submit a PR to deno itself to add CoffeeScript as another first-class citizen alongside TypeScript. I can imagine that might be a much more delicate proposal.

I’m a member of the Node.js modules working group trying to bring ES modules support to Node without breaking CommonJS (or CoffeeScript or TypeScript 😄). At some point Node will support ES modules, and when that happens I’m interested in having the CoffeeScript NPM package export an ES module version.

[jashkenas]

At some point Node will support ES modules, and when that happens I’m interested in having the CoffeeScript NPM package export an ES module version.

Very exciting, Geoffrey!

[shreeve]

@GeoffreyBooth - Does the new CoffeeScript 2.4.0 now support this ES module version? I think it does, based on your comments. How hard is it to produce a rollup-style minified version that could be put on a CDN?

[GeoffreyBooth]

It’s here: https://coffeescript.org/browser-compiler-modern/coffeescript.js

That’s hosted by GitHub Pages, so it doesn’t cost us anything, so you should feel free to link to it. I forgot to include that file in the bundle I published to NPM (d’oh) so it’ll be included there in 2.4.1 (#5186) whenever that gets released, and then the file will be accessible from unpkg and jsdelivr and other sources like that.

[shreeve]

Awesome! Thanks Geoffrey.

On Mon, Apr 1, 2019 at 4:32 PM Geoffrey Booth notifications@github.com wrote:

It’s here: https://coffeescript.org/browser-compiler-modern/coffeescript.js

That’s hosted by GitHub Pages, so it doesn’t cost us anything, so you should feel free to link to it. I forgot to include that file in the bundle I published to NPM (d’oh) so it’ll be included there in 2.4.1 (#5186 https://github.com/jashkenas/coffeescript/pull/5186) whenever that gets released, and then the file will be accessible from unpkg and jsdelivr and other sources like that.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jashkenas/coffeescript/issues/5150#issuecomment-478771163, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIuG5xP7f-kSOrJf6r9P8BtzBlqhmdYks5vcoj3gaJpZM4aIWxR .

[shreeve]

Do those services (unpkg, jsdelivr, etc) also minify? I think the file was 531k or so otherwise. Not sure if it’s something that should be run through rollupjs or something like that.

—

On Mon, Apr 1, 2019 at 4:34 PM Steve Shreeve steve.shreeve@gmail.com wrote:

Awesome! Thanks Geoffrey.

On Mon, Apr 1, 2019 at 4:32 PM Geoffrey Booth notifications@github.com wrote:

It’s here: https://coffeescript.org/browser-compiler-modern/coffeescript.js

That’s hosted by GitHub Pages, so it doesn’t cost us anything, so you should feel free to link to it. I forgot to include that file in the bundle I published to NPM (d’oh) so it’ll be included there in 2.4.1 (#5186 https://github.com/jashkenas/coffeescript/pull/5186) whenever that gets released, and then the file will be accessible from unpkg and jsdelivr and other sources like that.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jashkenas/coffeescript/issues/5150#issuecomment-478771163, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIuG5xP7f-kSOrJf6r9P8BtzBlqhmdYks5vcoj3gaJpZM4aIWxR .

[GeoffreyBooth]

I’m not aware that they minify. We could produce a minified version as part of our own build process; the non-ES module version of the browser compiler is minified, there’s no reason the ES version shouldn’t be. We’d need to make sure that the import and export statements are transpiled into CommonJS though.

Where would this be used where file size/download time would be a concern? The only places where I expect the browser compiler to be used are the CoffeeScript docs and sites like jsfiddle and codepen.

[kitsonk]

Hey! Just piping in on this issue. Contributor to Deno and look a lot at the compiler APIs. We have denoland/deno#1739. In order to get "good" CoffeeScript support, we would want to plug in the CoffeeScript compiler to the yet to be public compiler API. This would give the equivalent level of support that the TypeScript compiler has which includes the ability to fetch remote modules and have automatic caching.

We have most of the fundamentals almost there and I will be working on it in the near future. It is planned that the public API for compiler will run as a web worker. The options passed to the web worker would define what media types and extensions should be routed to the compiler when attempted to be loaded. Those will be sent to the compiler and the compiler is expected to return an ESM module which will then be injected into the runtime environment.

A compiler should have a single bundle of JavaScript which interfaces to the public API.

[shreeve]

It would be wonderful to have excellent (first class, on par with TypeScript) support for CoffeeScript in deno.

How far off is the public API for deno?

On Mon, Apr 1, 2019 at 5:18 PM Kitson Kelly notifications@github.com wrote:

Hey! Just piping in on this issue. Contributor to Deno and look a lot at the compiler APIs. We have denoland/deno#1739 https://github.com/denoland/deno/issues/1739. In order to get "good" CoffeeScript support, we would want to plug in the CoffeeScript compiler to the yet to be public compiler API. This would give the equivalent level of support that the TypeScript compiler has which includes the ability to fetch remote modules and have automatic caching.

We have most of the fundamentals almost there and I will be working on it in the near future. It is planned that the public API for compiler will run as a web worker. The options passed to the web worker would define what media types and extensions should be routed to the compiler when attempted to be loaded. Those will be sent to the compiler and the compiler is expected to return an ESM module which will then be injected into the runtime environment.

A compiler should have a single bundle of JavaScript which interfaces to the public API.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jashkenas/coffeescript/issues/5150#issuecomment-478781189, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIuG8A1Yx2S02x8HPyGXlMx4sy4pf-vks5vcpPAgaJpZM4aIWxR .

[GeoffreyBooth]

@kitsonk Thanks for chiming in! We appreciate the support.

CoffeeScript’s Node API today is described here: https://coffeescript.org/#nodejs-usage. Currently it takes a string (CoffeeScript source code) as input and returns an object (JavaScript source code, and source maps) as output. That’s it.

I’ve spent a little time working in the Meteor plugin for CoffeeScript; it wraps the CoffeeScript compiler in a class that handles caching and other niceties. Would your compiler API be like that? Or would the CoffeeScript compiler itself, or something that wraps it, be expected to handle caching? What about resolving URLs of import statements?

[kitsonk]

The privaliged side of Deno (Rust) would handle the module resolution and caching. I would suspect given the NodeJS API, that the wrapper would be super light. After a compiler is instantiated with something like this:

new Worker("https://coffeescript.org/browser-compiler-modern/coffeescript.js", {
  deno: {
    extensions: [ "coffee" ],
    media_types: [ "application/vnd.coffeescript" ]
  }
});

The web worker would receive an onmessage which would contain a payload of a Module Specifier and a Referrer. The compiler would need to use a privileged API to fetch the contents of module (fetchModuleMetaData) supplying the same module specifier and referrer. That would cause the privileged side (Rust) to resolve the module, including retrieving a remote module from the network, and returning the fully qualified module specifier, the media type, and any source code. If the compiler needs other modules meta data to fully transpile the code, it can continue to request additional modules (e.g. TypeScript has to resolve all dependencies in order to properly type check a module).

The compiler is then expected to pass back an ESM module and a source map.

The compiler will only be called to transpile a module if Rust has determined that either the cache is invalid or that the module isn't cached. When the data is returned, Rust will handle caching it locally.

When the module is returned, the privileged side will inject the module into the V8 runtime and V8 will resolve any additional dependencies, if those dependencies resolve to an extension or media type covered by the compiler, it will then repeat the cycle.

We haven't thought through the details of externally configuring a compiler yet, so that you would "always" have it at startup.

As far as a timeline, I have another piece of work in my backlog before really starting the public compiler API, and we need full web worker support which is about to land in Deno. So I expect I would really start working on it in a week or two. We always talked about using CoffeeScript as the "use case" for proving out the public compiler API, so this was timely.

[GeoffreyBooth]

@kitsonk Has there been any progress on the Deno public compiler API?

[kitsonk]

@GeoffreyBooth yes and no... as in we have been doing a lot of restructuring of the TypeScript compiler with an eye to have a public API for it, we have also have internally created handles based on media types and extensions, to support the concept of additional "types" being registered at runtime.

But there have been other internal restructures of how operations work between the privileged and non-privileged sides that have had an impact. I am in progress, yet again, of changing the API for the compiler to resolve some of the performance issues. Hopefully this is the last major iteration and then we can start to work towards making it public.

So thanks for the ping as a reminder of the interest. It is still firmly on the roadmap, but I will remind Ry of the larger interest that there is out there.

[GeoffreyBooth]

Thanks for the update. For what it’s worth, I’m on the Node modules team and we’ve been grappling with how to enable Node to treat .js files as ES modules without breaking backward compatibility (where all .js files are treated as CommonJS modules). See here, our solution is a package.json field called "type", for "type": "module" like browsers’ <script type="module">. Inelegant, perhaps, but more flexible than file extensions. That’s something you might want to consider as you finalize your design; I can see folks wanting to evaluate CommonJS .js files in Deno, to take advantage of the hundreds of thousands of such packages out there, and so you’ll need a way to register a loader for .js files that applies to only some such files. Along those lines, users will probably also need a way to tell Deno “these .ts files should use the TypeScript 2.9 compiler, while these .ts files should use TypeScript 3.6” (and ditto for .coffee and other types). Presumably if your API is robust enough, people will have ways of achieving all these goals.

[kitsonk]

Thanks for the thoughts.

One of the implicit design goals of Deno is to try not to utilise external meta data to dictate behaviours. Ultimately we would try to leverage code as configuration. Hopefully we can fully express the information needed to "route" compilations to the right compiler via extending the configuration passed to the web worker. A developer could choose to externalise that configuration as a JSON file which is read by the program and passed to the worker, but that would be a users decision.

Something that is sort of in the same arena is that we have adopted import-maps. It is external meta data, but it is based on a web standard, which that would override the implicit design goal of minimising external meta data.

That’s something you might want to consider as you finalize your design; I can see folks wanting to evaluate CommonJS .js files in Deno, to take advantage of the hundreds of thousands of such packages out there, and so you’ll need a way to register a loader for .js files that applies to only some such files.

Explicitly supporting CommonJS is a really complex issue, something I think we want to discourage at all costs, because seeing how much the Node.js community is struggling with that transition. We really want to avoid the sins of the father being revisited on the children. That being said, that problem is sort of solved already with turning on "checkJs" as part of a tsconfig.json, which then transpiles any source CommonJS or AMD .js module to an ESM module. But it is far more than supporting CommonJS, as it would also imply Node.js module resolution logic to take advantage of most packages that are on npm.

My personal opinion is that there is a long term path to tapping that Node.js ecosystem in a supportable way with Deno, which is to look at something like Pika (https://www.pika.dev/cdn), where the team there have already indicated intent to help serve up modules that work well under Deno, when Deno comes along to fetch them. This means that Deno itself doesn't have to worry itself about all that stuff.

Along those lines, users will probably also need a way to tell Deno “these .ts files should use the TypeScript 2.9 compiler, while these .ts files should use TypeScript 3.6”.

I think we expect there to be tight coupling between the version of Deno and the version of TypeScript, and be continually opinionated about . We only ever want to distribute one "built in compiler" and really discourage any fracturing of TypeScript compilations.

We seem to have had good luck with focusing on an MVP and iterating from there, so while it might be desired to have routing to a userland compiler of some matching to sources other than just the extension/media type, the MVP would need to at least do that and then support iteration from there. I feel that focusing on passing a configuration to the worker is the most flexible way to be able to deliver an MVP, but something that can be extended and iterated on as new functionality is evolved.

[GeoffreyBooth]

Hi @kitsonk, is there any update on the progress of making it possible for Deno to support CoffeeScript?

We’re about to release a new version of CoffeeScript that integrates with ESLint and Prettier. Someday soon I’d love to include Deno in the list of supported integrations 😄

[kitsonk]

@GeoffreyBooth it is still on the roadmap. We have made quite a bit of progress, a lot of the fundementals are there (like the ability to route different media-types/extensions to different compiler, ability to dynamically declare "ops" that allow communication between Rust and JavaScript). We still very much want to do it, it is a matter actually making it happen. I've got a couple of other biggies in my backlog at the moment, but the friendly ping certainly makes me consider it. I will think about it again, because it certainly helps show the flexibility of Deno.

[GeoffreyBooth]

Thanks @kitsonk. I’ve been going through old issues and cleaning them up, and I’m looking at this one and thinking, is there a reason to keep this open? As in, will there be anything required on the CoffeeScript side for Deno to support CoffeeScript? We already publish an ES module version of our compiler that works in browsers, and it should probably work as is in Deno. When the last bits of Node’s new ES modules support land in January, I plan to publish a version of the CoffeeScript NPM package that provides both Node CommonJS and Node ES module support. I’m happy to output a Deno-specific version if that would be beneficial, if Deno needs anything like a version that loads files using Deno’s filesystem APIs.

But assuming there isn’t anything on our side, or anything that you know of yet, perhaps it would be better to open this as an issue in whatever repo you’re working in? And we can track the progress there.

[GeoffreyBooth]

Hi @kitsonk, congratulations on shipping Deno 1.0! Any update on support for external compilers?

[kitsonk]

I know, I know... I keep this issue in my inbox just to keep it front of mind. I think the infrastructure is there now and pretty hardened. We should really revisit this.

[shadowrylander]

... May I ask as to the status of this, @kitsonk? 😅

[CTimmerman]

CoffeeScript already compiles to JS, so it could also compile to TS, ES, or whatever Deno prefers to fetch.

[kitsonk]

@shadowrylander it is still on the longish term roadmap... there have been other features that keep taking priority. I still think about it often though.

[shadowrylander]

@kitsonk Good enough for me! 😸

[gcxfd]

@kitsonk something like nodejs --experimental-loader

[bkuri]

@kitsonk I'd really appreciate an update on this if you have some time to spare! :wink:

[usrtax]

@kitsonk any update? bun now support import other script https://github.com/oven-sh/bun/releases/tag/bun-v0.1.11

Bun's runtime now supports a plugin API.

import and require .svelte, .vue, .yaml, .scss, .less and other file extensions that Bun doesn't implement a builtin loader for
Dynamically generate ESM & CJS modules
The API is loosely based on esbuild's plugin API.

This code snippet lets you import .mdx files in Bun:

import { plugin } from "bun";
import { renderToStaticMarkup } from "react-dom/server";

// Their esbuild plugin runs in Bun (without esbuild)
import mdx from "@mdx-js/esbuild";
plugin(mdx());

// Usage
import Foo from "./bar.mdx";
console.log(renderToStaticMarkup(<Foo />));