microsoft / TypeScript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
https://www.typescriptlang.org
Apache License 2.0
101.01k stars 12.48k forks source link

Support `.mjs` output #18442

Closed demurgos closed 3 years ago

demurgos commented 7 years ago

Experimental support for ES modules just landed in Node 8.5 (changelog, PR). Since ES modules have some parsing and semantic differences, Node decided to use the mjs extension for ES modules (while js is for the "script" target and commonjs modules).

The current Typescript version (2.5.2) supports ES modules emission but uses the js extension by default. It means that to use it with Node, a post-compilation step is required to change the extension from js to mjs. This adds undesirable complexity to use native ES modules support with Node.

A solution would be to add a compiler option to output *.mjs files when emitting ES modules.

Edit (2018-03-22): The propositions below are a bit outdated. I recommend reading the issue to see the progression. See this comment for my current proposition.

Notes:

demurgos commented 6 years ago

@jpike88 Node's ES modules are still experimental and may change. This discussion has some posts providing manual alternatives to generate .mjs files from .ts files. I don't understand what kind of linting you want.


Here are some updates for those following this issue:

jpike88 commented 6 years ago

I've been watching the node/modules repo for a week, and been reading their progress. It's coming along nicely.

At this point, is the process described in this gulpfile still the best thing to use in the meantime?

https://github.com/demurgos/turbo-gulp/blob/2c6173ec0d936ed1ebd06114a59bd40113ade2f2/src/lib/target-tasks/build-typescript.ts#L89-L107

saschanaz commented 6 years ago

esm package allows ES2015 modules without .mjs extension, it may be a workaround.

jpike88 commented 6 years ago

Ok that worked really well. Looks like I'm using that for now, thanks.

SMotaal commented 6 years ago

After months of waiting and meanwhile loosing all interest in TypeScript (due to obstacles like this) and after spending weeks trying to understand how difficult it would be to strip away TypeScript features from source files JIT because ultimately .d.ts files are a given requirement of a good package... Today I took a five minute break and decided to hack my workspace's tsserver to support ".mjs".

It took me 5 minutes and it worked for a silly 5 minute hack.

So this finally worked:

// last-hope.ts
  import {runner} from './runner'; // runner.mjs
  type runner = typeof runner; // inferred

Maybe 5 hours and I could get compiler to work, but the thing is I don't have the knowledge the TypeScript team experts have.

I was under the impression that it was a challenge, I took that for granted.

Ultimately, frustration and a silly thought of searching for ".js" and patching accordingly proved that it is doable, and sadly, it is just not a priority for TypeScript.

weswigham commented 6 years ago

The issue is in waiting for all (interop) behavior to be specified, not in any technical implementation hurdle. How a node esm module interacts with a node cjs module (and how we can identify either in the type system) is still being designed; we could be conservative and let you do nothing across module type bounds, but that would be an awful experience. Additionally, the mechanism for detecting the correct module type for a package represented by a .d.ts is nontrivial - we must assume that all declaration files that exist today actually represent cjs, and then introduce some new marker for, eg a .d.mts. Additionally, we don't want to ship something and need to change it in a month or two when behaviors are finalized (and retain support for the guesswork for awhile), so we're watching development closely and when stuff comes to consensus and actually lands (and no, an experimental flag doesn't count), we'll start shipping something; but not before (and we're in the node working group looking at this, so we know it is being worked out). This matters more for us than, eg, Babel because while the emit is unlikely to change much, the typechecking behavior and expected import shape is way more up in the air, which for us can be breaking.

SMotaal commented 6 years ago

@weswigham... I cannot disagree with you about fully fledged support, but what I think is going unheard by everyone upstream is that people have been waiting since 2015 (not singling anyone specifically) and every time we take a step forward, someone else upstream takes ten steps back.

Those of us who are interested in experimental mjs are compromising because Node.js made this call, then compromising and using --experimental-module flag, so we are completely understanding of the reality and do not expect TypeScript to have had a ready strategy for mainstream (non-experimental) support.

So now let's switch perspectives, even after we compromised, we only asked that TypeScript offer a similar --experimental-modules mode (with all manner of disclaimers that would void any legal burdens from you guys) we just want what ever comes naturally or breaks naturally to happen when using this mode.

For us, this is the difference between helping you pinpoint areas that can help you with your progress, versus us having to stop working (experimenting) and instead waste months exploring possible "temporary" tooling to mediate the gaps or look into alternatives.

For at least myself, I did both, and considering all the other upstream stops since ES2015's ratification, and honestly because of the extent to which I was depending on TypeScript (because it never left me hanging), this issue right here has been the most detrimental to my productivity by far.

So back then in September or even January, I would have been able to be a good TypeScript citizen by uncovering potentially uncharted issues through experimental Extension.Mjs support. But now that I realize how depending on one tool can be detrimental - unless we are always in agreement - and hopefully with a "true" dialog taking place - I doubt that I will ever growing too comfortable with any one tool again - sadly even TypeScript.

Please appreciate that this is as honest and sincere a customer review as you can expect in such a forum.

saschanaz commented 6 years ago

Node.js ES module support is still experimental and requires a flag. Supporting it is not simple, even import { readFile } from "fs" won't work because Node.js gathers everything into default for CJS modules. I think using esm is the best before it gets stable enough to earn a good support from other tools like TypeScript.

SMotaal commented 6 years ago

@saschanaz no question, you are absolutely right for use cases where using esm appropriate. The reality of this option is that it is compiling your ESM module to CommonJS. This works great for most mainstream use, but excludes being able to specifically test experimental TypeScript scenarios with real modules, which is the who purpose an experimental stage (where users can actually be contributing in places where developers might not have been looking).

SMotaal commented 6 years ago

Since my last post (timestamped 4 hours ago now) I cloned TypeScript and did my best to take my experiment from hacking the compiled tsserver.js into the actual source.

This required very few changes than anticipated: (some changes are simply due to whitespace reformat/reflow)

 src/compiler/core.ts                  | 12 ++++++++----
 src/compiler/emitter.ts               |  3 ++-
 src/compiler/moduleNameResolver.ts    |  6 +++---
 src/compiler/program.ts               |  1 +
 src/compiler/resolutionCache.ts       |  2 +-
 src/compiler/types.ts                 |  1 +
 src/services/codefixes/importFixes.ts |  2 +-
 7 files changed, 17 insertions(+), 10 deletions(-)

Then for tests:

 src/harness/harness.ts                         | 18 ++++++++++++++----
 src/harness/projectsRunner.ts                  |  6 +++++-
 src/harness/unittests/reuseProgramStructure.ts |  8 ++++++++
 src/harness/unittests/tsserverProjectSystem.ts |  1 +
 4 files changed, 28 insertions(+), 5 deletions(-)

And testing: (needed a jake baseline-accept after eliminating other errors)

Discovered 103 unittest suites.
Discovering runner-based tests...
Discovered 12150 test files in 347ms.
Starting to run tests using 8 threads...
Batching initial test lists...
Batched into 8 groups with approximate total time of 2m6s in each group. (90.0% of total tests batched)

  [▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬] ✓ 60367 passing (2m31s)

  60367 passing (3m)

Linting: node node_modules/tslint/bin/tslint --project scripts/tslint/tsconfig.json --formatters-dir ./built/local/tslint/formatters --format autolinkableStylish
rm -rf tests/baselines/local/projectOutput/
Linting: node node_modules/tslint/bin/tslint --project src/tsconfig-base.json --formatters-dir ./built/local/tslint/formatters --format autolinkableStylish
✨  Done in 206.69s.

I will need to do some due diligence in the morning then open a PR so others with similar goals can collaborate on this.

SMotaal commented 6 years ago

The PR passed all tests. Since it is only a PR, the actual built files are compiled into built/local. If you are interested in using it, I am putting together a wrapper package and working on a README here:

https://gist.github.com/smotaal/4f9458cf43cd7a55a12d5c476c15400e

que-etc commented 6 years ago

@smotaal Have you considered implementing a node loader hook, which would natively load .js files as ES modules?

Here is a working draft:

// @filename: loader.mjs
export async function resolve(specifier, parentModuleURL, defaultResolver) {
    const resolved = defaultResolver(specifier, parentModuleURL);
    const {url} = resolved;

    /**
     * Implement you own criteria here. For instance, you could use a glob pattern:
     * src/**\/*\/*.js - consider all js files in the "src" folder to be ES modules.
     */
    if (resolved.format === 'cjs' && url.endsWith('.esm.js')) {
        // Tell node to load this module as ES module even though it doesn't
        // have the "mjs" extension.
        return {url, format: 'esm'};
    }

    return resolved;
}

// @filename: foo.esm.js
import bar from './bar.esm'

// @filename: bar.esm.js
export default 42;

// Usage: node --experimental-modules --loader ./loader.mjs foo.esm.js

This way you can avoid patching TypeScript and it works without any third party libraries.

SMotaal commented 6 years ago

Certainly, but performance and complexity makes it ideal for certain applications early on but not as requirements started to deal with integrations with other packages.

que-etc commented 6 years ago

Yeah, it's meant to be used only inside of an application. In case of a library, you'd definitely need to change the extension back to .mjs before publishing.

But anyway, with the former approach, integration with other packages is my biggest concern as well.

ctsstc commented 6 years ago

The future cannot be soon enough. Thanks for all the hard work on this. It definitely is hard to commit to something when it's "experimental".

gfmio commented 6 years ago

I have been needing this functionality for my own projects for quite a while. I had a script that would do the necessary transformations after the output had been generated, but it was a bit brittle, so I decided to fork the typescript package and patch it to support the generation of .mjs files (Github, NPM).

The patch itself was surprisingly easy since I only had to add a compiler option and the new extension type (Commit)

You can find my patched version as ts-mjs on NPM.

With the package installed as a dev dependency, you can now call the patched version of tsc as tsc-mjs and if you can add the compiler option --mjs to emit .mjs files instead of .js files.

If you use the --sourceMap compiler option, this will also emit the corresponding .mjs.map source map for your .mjs file.

If the output target for JSX would be preserve via one of the compiler options, regular JSX code with the regular extension .jsx will be emitted.

For example, if you run

tsc --sourceMap --declaration --target es3 --module commonjs index.ts

this will emit

index.d.ts (the declaration file)
index.js (the CommonJS ES3 module)
index.js.map (the source map for the CommonJS module)

If you run

tsc-mjs --sourceMap --declaration --target esnext --module esnext --mjs index.ts
index.d.ts (the declaration file)
index.mjs (the ES module)
index.mjs.map (the source map for the ES module)

The declaration files that are generated will be identical in both cases.

You can use this to publish hybrid modules by running both commands in succession (or in parallel, since they won't interfere with each other). The output will be:

index.d.ts (the declaration file)
index.js (the CommonJS ES3 module)
index.js.map (the source map for the CommonJS module)
index.mjs (the ES module)
index.mjs.map (the source map for the ES module)

If you have other source files, the corresponding 5 files will be generated for them.

If someone enters your package now via an .mjs file, node will automatically search for .mjs file in all import (or require) statements (and it will fall back to .js should an .mjs file not exist).

In your package.json, you can make the main field "dynamic" by removing the extension of the linked file.

{
    "main": "index",
    "browser": "index.js",
    "module": "index.mjs",
    "types": "index.d.ts"
}

For a real-life example of this being in use, check out my hsluv-ts library (Github, NPM).

(Note: The link to that NPM package will be broken during the next 24 hours, because I made a mistake while publishing, had to unpublish it and now I have to wait for 24 hours to republish it).


This should be straightforward to integrate into the mainline compiler, if desired, since the patch is so small. It just comes with the addition of one compiler option and it makes everyone's life a lot easier. Let me know what your thoughts are and I'm happy to create a pull request!

-@gfmio

weswigham commented 6 years ago

This is your daily reminder that .mjs and how cjs/esm interop works in node is unstable and liable to change until it ships unflagged - you should not use it in a production workflow unless you're amenable to significant churn. 💓

We will support it in TS once it's stable - use what you find at your own risk. 😃

Jamesernator commented 6 years ago

@weswigham I would like to point out that regardless of what Node.js decides to do it's already possible to use any extension in the browser. Which I think is important.

Where I work most of the code is still primarily classic scripts, and I've been introducing bits of modules in internal tools, but to prevent tooling break from new files that are modules and to minimize any nightmarish configuration .mjs was a natural pick. NOTE that this is primarily browser based code (but there are some node tools too using @std/esm) and I think it would be good for TypeScript to support an extension that's guaranteed to be a module for this reason.

EDIT: Just to clarify the actual extension doesn't matter, but adopting modules into an existing code base of primarily classic scripts without breaking tooling is just difficult without a different extension. It's not necessarily important to support every extension but having at least one that's guaranteed to be a module source text rather than classic script makes introducing them into existing codebases a lot easier.

demurgos commented 6 years ago

Regarding Typescript and Node's native ESM, there is an issue in nodejs/modules. It's relatively old but it wasn't posted here.

Changing the file extension of the output file is one of the least problems. On the emit side, TS tries to keep the differences minimal between the output using different module types. Node having different mechanism when consuming CJS or ESM may be a hard difference to hide. The biggest issue though is consuming ESM and properly typechecking the dependencies. The type declaration files are mostly using ES syntax, even if the module is actually written in CJS (for example, it "lies" about available named exports). When using esModuleInterop, Typescript cannot know if the import statement is actually valid (ES namespace or default export). If you throw in dual builds and import-based resolution (for example .js/.mjs or main/module), the situation quickly becomes complicated: the type of the dependency depends on how you consume it.


This also a reminder to watch the nodejs/modules repo. The discussions may be hard to follow, but at least they're in a single place.


I'm currently looking into code coverage for native ESM in Node. It looks promising but is still experimental. I'll report back when things settle a bit.

romap0 commented 5 years ago

Any news on it?

saschanaz commented 5 years ago

@romap0 TC39 proposed an alternative: https://github.com/tc39/proposal-modules-pragma

TypeScript will probably follow TC39 when a consensus happens.

jpike88 commented 5 years ago

I bit the bullet and used https://github.com/fuse-box/fuse-box works good

demurgos commented 5 years ago

@saschanaz My understanding is the the module pragma has been rejected.

@jpike88 Isn't fuse-box a bundler? I am not sure how this relevant to TS emitting .mjs (which is mostly a Node concern).

@romap0 We are still waiting for Node to settle on a design for ESM support. Discussions are advancing on the nodejs/modules repo.

saschanaz commented 5 years ago

@demurgos It's still in stage 1 proposal list: https://github.com/tc39/proposals#stage-1

jpike88 commented 5 years ago

I use fuse-box for my server code too (i also noticed it initialises a lot faster than the .mjs method, likely due to sub-optimal linking in Node itself)

demurgos commented 5 years ago

@saschanaz It's been in stage 1 for a few years now. It has been discussed at TC39 meetings and rejected. Have there been TC39 meetings discussing it again in the last months?

On the TC39 side, the most recent development (that I am aware of) has been around maybe adding a new module record type for interop with CJS.

saschanaz commented 5 years ago

It's added to the list 3 months ago so I would say it's not "rejected", as any rejected proposals instead go to "inactive proposals" list.

revmischa commented 5 years ago

My use case: developing a MJS/ES2015 module and importing it into another project. While I develop, I am running tsc -w. If a post-compilation step is needed to rename the output file to .mjs then how I can develop a project that depends on my library being recompiled with tsc -w?

Not a rhetorical question, I don't get it. Isn't this how people normally write modules? Have another project that imports the module (via yarn link) and compile both projects in --watch mode? How can this work with ES2015 modules if a post-compilation step is required to rename a file from .js to .mjs? Am I missing something?

phil-lgr commented 5 years ago

Sorry to add noise in the thread, maybe people will find this useful:

https://github.com/kevinpollet/typescript-es-modules-node-example/tree/master

the gist of it:

  1. "type": "module", in package.json
  2. in tsconfig.json: "module": "esnext", and "target": "esnext",
  3. run it:
    node  --experimental-modules --es-module-specifier-resolution=node dist/server/server.js
Nainterceptor commented 5 years ago

Hello,

If someone like me come here, want to use rename but want to fix .mjs.map :

{
  "scripts": {
    // Out is dist/esm
    "build:esm": "tsc -p config/tsconfig.esm.json && npm-s build:esm:rename build:esm:edit-map",
    "build:esm:edit-map": "for FILE in `find dist -type f -name \"*.mjs\"`; do json -I -f $FILE.map -e \"this.file='$(basename -- $FILE)'\" && sed 's/sourceMappingURL=\\([^ ]*\\).js.map/sourceMappingURL=\\1.mjs.map/g' $FILE > $FILE.remap && mv $FILE.remap $FILE ; done",
    "build:esm:rename": "renamer --find .js --replace .mjs 'dist/esm/**/*.js*'",
  },

  "devDependencies": {
    // ...
    "json": "^9.0.6",
    "npm-run-all": "^4.1.5",
    "renamer": "^1.1.3"
  },
}

Note also, that main vs module key in package.json is ignored by nodeJS (ref)

SomethingSexy commented 5 years ago

@phil-lgr that actually works pretty well. The one place I am stuck though is in SSR projects where we have some code that is pulling in CSS files still (for webpack). We are using ignore-styles package but that doesn't seem to work with this new system.

demurgos commented 4 years ago

Node.js support for unflagged native ESM should land in Node 13.2.0 (probably next week). Here's the PR for the announcement and details.

SMotaal commented 4 years ago

@weswigham Are there any updates related to this? Would be greatly appreciated to have this coincide with unflagging if there are updates.

SMotaal commented 4 years ago

@SomethingSexy Assuming you are interested in running the ESM files in Node.js, you can explore --loader which is still experimental but can help deal with things like CSS.

In most cases, I'd urge people to only do experimental things when experimenting, and in this case it is still true, but CSS files loaded in this context are likely merely noop and no error, right?

If that is the case, I'd recommend exploring it cautiously — see Experimental Loader hooks.

Important — Please note that experimental and/or flagged features are highly discouraged if you are publishing to NPM… etc.

It is fair to mention that browsers have looked into non-ESM imports a bit but backtracked (I believe it was JSON) which is discouraging at least for anyone planning on doing away with bundling and wanting all the mojo.

SomethingSexy commented 4 years ago

@SMotaal that is a good idea, I will look into that option for our testing. Ya in this case these files are noop.

SMotaal commented 4 years ago

@SomethingSexy Can you update your response with a link to this pattern once/if you have it public.

xiaoxiangmoe commented 4 years ago

So node 13.2 has been released with .mjs support without flag: https://github.com/nodejs/node/releases/tag/v13.2.0

Can we config mjs and cjs output?

dandv commented 4 years ago

What's the TL;DR for generating .mjs from TypeScript files, now that it's been a couple months since .mjs support was unflagged in Node?

@phil-lgr's tsconfig.json does most of it, though it's a bit unclear and outdated, so here's what you need to do as of Feb 2020:

  1. In tsconfig.json: "target": "esnext" and "module": "esnext". This will output .js files still, not .mjs.
  2. In package.json, add "type": "module", so Node can execute the generated .js files with import statements.
  3. Run Node with the --experimental-specifier-resolution=node flag, so you don't have to specify the .js extension in the import statements.
frank-dspeed commented 4 years ago

@dandv i my self use rollup to overcome that limitations it has a typescript plugin a checking and a none checking one it has options to define the output filenames.

jkrems commented 4 years ago

Another option is to use tsc for type checking only and to use babel to write proper module ouputs:

Using that approach, you'd still use tsc with emitDeclarationOnly to generate type definitions. But the generated JavaScript will be in proper module files and have fully qualified relative specifiers.

There's still rough edges with this since TypeScript may not properly recognize third party imports from modules but it may be a helpful workaround.

frank-dspeed commented 4 years ago

the best solution is to avoid .ts extension and go for .js extension then marking the source as module via adding type: module to a package.json near the module imports or even in the package root

then add JSDOC Annotations and turn checkJs true in tsconfig.json

SMotaal commented 4 years ago

@frank-dspeed I know, once upon a time when some said use ".ts"… now when it is fair to want to do ".js"… things take time I guess, and projects especially like to keep at it thinking it will be solved upstream, even if it has already.

So for folks on this thread, please research closer, TypeScript is a development tool, ".ts" is the way it was best accomplished at one point, but please don't take anyone's word for it, do your own digging. In the end, TypeScript is not going anywhere, even if you use ".js" you are still getting the best perks it has to offer today.

@demurgos Fair to note… this issue likely needs to be renamed to include "cjs" and/or "mjs", which can be achieved with something like outputFormats which would (string|{format: string, extension?: string})[].

frank-dspeed commented 4 years ago

@SMotaal assert{} was always superior + jsdoc.

I do not really care for typescript itself at present tsserver is the backbone of vs code that makes it matter not that it is a tool.

SMotaal commented 4 years ago

@frank-dspeed so I feel a little confused when addressing technical issues, how we sometimes mistake what is absolutely everyone's right to express (ie opinions and preferences) with what we all mutually must not dismiss on account of our own emotions on the former (ie works for everyone, not just the "selfish" me) — the whole idea of open source looking like social media sadly makes this kind of knee jerk reaction likely for anyone of us…

@SamHH @csvn — it really helps if you articulate where you disagree, please, I am inclined to venture but I do not want to be dismissive, so if the 👎 is not meant to discount the very real pain that comes from how typescript and vscode are coupled, which is really the technical point of relevance here imho, then articulating would really help everyone involved… thanks :)

csvn commented 4 years ago

@SMotaal I disagreed mostly with this statement:

TypeScript is not going anywhere, even if you use ".js" you are still getting the best perks it has to offer today.

I use Typescript in almost all projects I work on, and I always use strict: true. Whenever I've used .js and JSDoc (e.g. for various smaller scripts) the experience I've had has always been vastly inferior to using .ts. A very subjective note, but I feel readability suffers a lot when you need to add comments inside methods and next to variable assignments.

I do wish Typescript supported .mjs so it was easier to use native ESM with Node v13 for experimenting, but I don't feel like using .js is a great option for anything but very small projects. 🤷‍♂️

SMotaal commented 4 years ago

I don't see where we disagree honestly, yes, .mjs from .ts and .js, this is fundamentally necessary for the ecosystem, but now even .cjs needs to become factored in the whole process.

@csvn it might add context to add I've used TypeScript for ages, and I know exactly the trade-offs, why I moved to .js files and still use TypeScript even as a compiler.

I hope you can appreciate at least the context of my comment in this thread alone, but I do not think we are arguing differently on the core ask in this op, fair?

frank-dspeed commented 4 years ago

@csvn i found out it works best when defining typescript like interfaces via @typedef then you can use short /* @type {name} / near the var so it is less verbose. you can also do @type {import('./@types....').name}

TheMrZZ commented 4 years ago

Any news on that functionality ?

frank-dspeed commented 4 years ago

@TheMrZZ you can expect this to stay around for a longer period of time 2+ Years maybe more. I was trying to do a fast fork to fix that it was not so easy it would take some weeks of fulltime work so we all are doomed.

LinusU commented 4 years ago

If anyone is interested on how to configure TypeScript & Node.js 14 to work together, I just made a post on StackOverflow on how to do that here: https://stackoverflow.com/a/61305579/148072

tl;dr