Closed demurgos closed 3 years ago
@jpike88 Node's ES modules are still experimental and may change. This discussion has some posts providing manual alternatives to generate .mjs
files from .ts
files. I don't understand what kind of linting you want.
Here are some updates for those following this issue:
Dynamic imports landed on Node stable. I tested it with Node 9.8 but I think it's there since 9.6. There is still no import.meta
but it's already pretty good. I haven't played with it much yet. I think that setting "module" to es2017
or esnext
should be enough to emit code using it. I don't know what is emitted for older module versions or how type definitions are handled.
Babel has an issue to change its defaults for module types (it's an interesting read). There's a comment by @weswigham about Typescript and .mjs
:
In the browser,
.js
is king (and the default), and in node, tons of effort is (actively) being poured into making .js as close to the default as possible (mode flags, loaders, etc - and still not finalized). I believe it's premature to call.mjs
acceptable as the default mode of operation for any transpiler (I know we're holding off on it at typescript because 1. we really dislike.mjs
since it'll cause our resolver complexity to bloom considerably and seriously damage the ecosystem (many definitions assume babel-style transparent cjs interop), and 2. there's the possibility that its usage ends up really low and isn't worth supporting if it's not implemented well, given the large existing cjs ecosystem) prior to any implementation that actually uses.mjs
shipping without anexperimental
flag.
https://github.com/babel/babel/pull/7501#issuecomment-370636191
I recommend watching the nodejs/modules repo
I've been watching the node/modules repo for a week, and been reading their progress. It's coming along nicely.
At this point, is the process described in this gulpfile still the best thing to use in the meantime?
esm
package allows ES2015 modules without .mjs
extension, it may be a workaround.
Ok that worked really well. Looks like I'm using that for now, thanks.
After months of waiting and meanwhile loosing all interest in TypeScript (due to obstacles like this) and after spending weeks trying to understand how difficult it would be to strip away TypeScript features from source files JIT because ultimately .d.ts files are a given requirement of a good package... Today I took a five minute break and decided to hack my workspace's tsserver to support ".mjs".
It took me 5 minutes and it worked for a silly 5 minute hack.
So this finally worked:
// last-hope.ts
import {runner} from './runner'; // runner.mjs
type runner = typeof runner; // inferred
Maybe 5 hours and I could get compiler to work, but the thing is I don't have the knowledge the TypeScript team experts have.
I was under the impression that it was a challenge, I took that for granted.
Ultimately, frustration and a silly thought of searching for ".js" and patching accordingly proved that it is doable, and sadly, it is just not a priority for TypeScript.
The issue is in waiting for all (interop) behavior to be specified, not in any technical implementation hurdle. How a node esm module interacts with a node cjs module (and how we can identify either in the type system) is still being designed; we could be conservative and let you do nothing across module type bounds, but that would be an awful experience. Additionally, the mechanism for detecting the correct module type for a package represented by a .d.ts
is nontrivial - we must assume that all declaration files that exist today actually represent cjs, and then introduce some new marker for, eg a .d.mts
. Additionally, we don't want to ship something and need to change it in a month or two when behaviors are finalized (and retain support for the guesswork for awhile), so we're watching development closely and when stuff comes to consensus and actually lands (and no, an experimental flag doesn't count), we'll start shipping something; but not before (and we're in the node working group looking at this, so we know it is being worked out). This matters more for us than, eg, Babel because while the emit is unlikely to change much, the typechecking behavior and expected import shape is way more up in the air, which for us can be breaking.
@weswigham... I cannot disagree with you about fully fledged support, but what I think is going unheard by everyone upstream is that people have been waiting since 2015 (not singling anyone specifically) and every time we take a step forward, someone else upstream takes ten steps back.
Those of us who are interested in experimental mjs
are compromising because Node.js made this call, then compromising and using --experimental-module flag, so we are completely understanding of the reality and do not expect TypeScript to have had a ready strategy for mainstream (non-experimental) support.
So now let's switch perspectives, even after we compromised, we only asked that TypeScript offer a similar --experimental-modules mode (with all manner of disclaimers that would void any legal burdens from you guys) we just want what ever comes naturally or breaks naturally to happen when using this mode.
For us, this is the difference between helping you pinpoint areas that can help you with your progress, versus us having to stop working (experimenting) and instead waste months exploring possible "temporary" tooling to mediate the gaps or look into alternatives.
For at least myself, I did both, and considering all the other upstream stops since ES2015's ratification, and honestly because of the extent to which I was depending on TypeScript (because it never left me hanging), this issue right here has been the most detrimental to my productivity by far.
So back then in September or even January, I would have been able to be a good TypeScript citizen by uncovering potentially uncharted issues through experimental Extension.Mjs support. But now that I realize how depending on one tool can be detrimental - unless we are always in agreement - and hopefully with a "true" dialog taking place - I doubt that I will ever growing too comfortable with any one tool again - sadly even TypeScript.
Please appreciate that this is as honest and sincere a customer review as you can expect in such a forum.
Node.js ES module support is still experimental and requires a flag. Supporting it is not simple, even import { readFile } from "fs"
won't work because Node.js gathers everything into default
for CJS modules. I think using esm
is the best before it gets stable enough to earn a good support from other tools like TypeScript.
@saschanaz no question, you are absolutely right for use cases where using esm appropriate. The reality of this option is that it is compiling your ESM module to CommonJS. This works great for most mainstream use, but excludes being able to specifically test experimental TypeScript scenarios with real modules, which is the who purpose an experimental stage (where users can actually be contributing in places where developers might not have been looking).
Since my last post (timestamped 4 hours ago now) I cloned TypeScript and did my best to take my experiment from hacking the compiled tsserver.js into the actual source.
This required very few changes than anticipated: (some changes are simply due to whitespace reformat/reflow)
src/compiler/core.ts | 12 ++++++++----
src/compiler/emitter.ts | 3 ++-
src/compiler/moduleNameResolver.ts | 6 +++---
src/compiler/program.ts | 1 +
src/compiler/resolutionCache.ts | 2 +-
src/compiler/types.ts | 1 +
src/services/codefixes/importFixes.ts | 2 +-
7 files changed, 17 insertions(+), 10 deletions(-)
Then for tests:
src/harness/harness.ts | 18 ++++++++++++++----
src/harness/projectsRunner.ts | 6 +++++-
src/harness/unittests/reuseProgramStructure.ts | 8 ++++++++
src/harness/unittests/tsserverProjectSystem.ts | 1 +
4 files changed, 28 insertions(+), 5 deletions(-)
And testing: (needed a jake baseline-accept
after eliminating other errors)
Discovered 103 unittest suites.
Discovering runner-based tests...
Discovered 12150 test files in 347ms.
Starting to run tests using 8 threads...
Batching initial test lists...
Batched into 8 groups with approximate total time of 2m6s in each group. (90.0% of total tests batched)
[▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬] ✓ 60367 passing (2m31s)
60367 passing (3m)
Linting: node node_modules/tslint/bin/tslint --project scripts/tslint/tsconfig.json --formatters-dir ./built/local/tslint/formatters --format autolinkableStylish
rm -rf tests/baselines/local/projectOutput/
Linting: node node_modules/tslint/bin/tslint --project src/tsconfig-base.json --formatters-dir ./built/local/tslint/formatters --format autolinkableStylish
✨ Done in 206.69s.
I will need to do some due diligence in the morning then open a PR so others with similar goals can collaborate on this.
The PR passed all tests. Since it is only a PR, the actual built files are compiled into built/local
. If you are interested in using it, I am putting together a wrapper package and working on a README here:
https://gist.github.com/smotaal/4f9458cf43cd7a55a12d5c476c15400e
@smotaal
Have you considered implementing a node loader hook, which would natively load .js
files as ES modules?
Here is a working draft:
// @filename: loader.mjs
export async function resolve(specifier, parentModuleURL, defaultResolver) {
const resolved = defaultResolver(specifier, parentModuleURL);
const {url} = resolved;
/**
* Implement you own criteria here. For instance, you could use a glob pattern:
* src/**\/*\/*.js - consider all js files in the "src" folder to be ES modules.
*/
if (resolved.format === 'cjs' && url.endsWith('.esm.js')) {
// Tell node to load this module as ES module even though it doesn't
// have the "mjs" extension.
return {url, format: 'esm'};
}
return resolved;
}
// @filename: foo.esm.js
import bar from './bar.esm'
// @filename: bar.esm.js
export default 42;
// Usage: node --experimental-modules --loader ./loader.mjs foo.esm.js
This way you can avoid patching TypeScript and it works without any third party libraries.
Certainly, but performance and complexity makes it ideal for certain applications early on but not as requirements started to deal with integrations with other packages.
Yeah, it's meant to be used only inside of an application. In case of a library, you'd definitely need to change the extension back to .mjs
before publishing.
But anyway, with the former approach, integration with other packages is my biggest concern as well.
The future cannot be soon enough. Thanks for all the hard work on this. It definitely is hard to commit to something when it's "experimental".
I have been needing this functionality for my own projects for quite a while. I had a script that would do the necessary transformations after the output had been generated, but it was a bit brittle, so I decided to fork the typescript
package and patch it to support the generation of .mjs
files (Github, NPM).
The patch itself was surprisingly easy since I only had to add a compiler option and the new extension type (Commit)
You can find my patched version as ts-mjs
on NPM.
With the package installed as a dev dependency, you can now call the patched version of tsc
as tsc-mjs
and if you can add the compiler option --mjs
to emit .mjs
files instead of .js
files.
If you use the --sourceMap
compiler option, this will also emit the corresponding .mjs.map
source map for your .mjs
file.
If the output target for JSX would be preserve via one of the compiler options, regular JSX code with the regular extension .jsx
will be emitted.
For example, if you run
tsc --sourceMap --declaration --target es3 --module commonjs index.ts
this will emit
index.d.ts (the declaration file)
index.js (the CommonJS ES3 module)
index.js.map (the source map for the CommonJS module)
If you run
tsc-mjs --sourceMap --declaration --target esnext --module esnext --mjs index.ts
index.d.ts (the declaration file)
index.mjs (the ES module)
index.mjs.map (the source map for the ES module)
The declaration files that are generated will be identical in both cases.
You can use this to publish hybrid modules by running both commands in succession (or in parallel, since they won't interfere with each other). The output will be:
index.d.ts (the declaration file)
index.js (the CommonJS ES3 module)
index.js.map (the source map for the CommonJS module)
index.mjs (the ES module)
index.mjs.map (the source map for the ES module)
If you have other source files, the corresponding 5 files will be generated for them.
If someone enters your package now via an .mjs
file, node will automatically search for .mjs
file in all import
(or require
) statements (and it will fall back to .js
should an .mjs
file not exist).
In your package.json
, you can make the main
field "dynamic" by removing the extension of the linked file.
{
"main": "index",
"browser": "index.js",
"module": "index.mjs",
"types": "index.d.ts"
}
For a real-life example of this being in use, check out my hsluv-ts
library (Github, NPM).
(Note: The link to that NPM package will be broken during the next 24 hours, because I made a mistake while publishing, had to unpublish it and now I have to wait for 24 hours to republish it).
This should be straightforward to integrate into the mainline compiler, if desired, since the patch is so small. It just comes with the addition of one compiler option and it makes everyone's life a lot easier. Let me know what your thoughts are and I'm happy to create a pull request!
-@gfmio
This is your daily reminder that .mjs
and how cjs/esm interop works in node is unstable and liable to change until it ships unflagged - you should not use it in a production workflow unless you're amenable to significant churn. 💓
We will support it in TS once it's stable - use what you find at your own risk. 😃
@weswigham I would like to point out that regardless of what Node.js decides to do it's already possible to use any extension in the browser. Which I think is important.
Where I work most of the code is still primarily classic scripts, and I've been introducing bits of modules in internal tools, but to prevent tooling break from new files that are modules and to minimize any nightmarish configuration .mjs
was a natural pick. NOTE that this is primarily browser based code (but there are some node tools too using @std/esm
) and I think it would be good for TypeScript to support an extension that's guaranteed to be a module for this reason.
EDIT: Just to clarify the actual extension doesn't matter, but adopting modules into an existing code base of primarily classic scripts without breaking tooling is just difficult without a different extension. It's not necessarily important to support every extension but having at least one that's guaranteed to be a module source text rather than classic script makes introducing them into existing codebases a lot easier.
Regarding Typescript and Node's native ESM, there is an issue in nodejs/modules
. It's relatively old but it wasn't posted here.
Changing the file extension of the output file is one of the least problems. On the emit side, TS tries to keep the differences minimal between the output using different module types. Node having different mechanism when consuming CJS or ESM may be a hard difference to hide.
The biggest issue though is consuming ESM and properly typechecking the dependencies. The type declaration files are mostly using ES syntax, even if the module is actually written in CJS (for example, it "lies" about available named exports). When using esModuleInterop
, Typescript cannot know if the import statement is actually valid (ES namespace or default
export). If you throw in dual builds and import-based resolution (for example .js
/.mjs
or main
/module
), the situation quickly becomes complicated: the type of the dependency depends on how you consume it.
This also a reminder to watch the nodejs/modules
repo. The discussions may be hard to follow, but at least they're in a single place.
I'm currently looking into code coverage for native ESM in Node. It looks promising but is still experimental. I'll report back when things settle a bit.
Any news on it?
@romap0 TC39 proposed an alternative: https://github.com/tc39/proposal-modules-pragma
TypeScript will probably follow TC39 when a consensus happens.
I bit the bullet and used https://github.com/fuse-box/fuse-box works good
@saschanaz My understanding is the the module pragma has been rejected.
@jpike88
Isn't fuse-box
a bundler? I am not sure how this relevant to TS emitting .mjs
(which is mostly a Node concern).
@romap0 We are still waiting for Node to settle on a design for ESM support. Discussions are advancing on the nodejs/modules repo.
@demurgos It's still in stage 1 proposal list: https://github.com/tc39/proposals#stage-1
I use fuse-box for my server code too (i also noticed it initialises a lot faster than the .mjs method, likely due to sub-optimal linking in Node itself)
@saschanaz It's been in stage 1 for a few years now. It has been discussed at TC39 meetings and rejected. Have there been TC39 meetings discussing it again in the last months?
On the TC39 side, the most recent development (that I am aware of) has been around maybe adding a new module record type for interop with CJS.
It's added to the list 3 months ago so I would say it's not "rejected", as any rejected proposals instead go to "inactive proposals" list.
My use case: developing a MJS/ES2015 module and importing it into another project. While I develop, I am running tsc -w
. If a post-compilation step is needed to rename the output file to .mjs
then how I can develop a project that depends on my library being recompiled with tsc -w
?
Not a rhetorical question, I don't get it.
Isn't this how people normally write modules? Have another project that imports the module (via yarn link
) and compile both projects in --watch
mode? How can this work with ES2015 modules if a post-compilation step is required to rename a file from .js
to .mjs
? Am I missing something?
Sorry to add noise in the thread, maybe people will find this useful:
https://github.com/kevinpollet/typescript-es-modules-node-example/tree/master
the gist of it:
"type": "module",
in package.json"module": "esnext",
and "target": "esnext",
node --experimental-modules --es-module-specifier-resolution=node dist/server/server.js
Hello,
If someone like me come here, want to use rename but want to fix .mjs.map :
{
"scripts": {
// Out is dist/esm
"build:esm": "tsc -p config/tsconfig.esm.json && npm-s build:esm:rename build:esm:edit-map",
"build:esm:edit-map": "for FILE in `find dist -type f -name \"*.mjs\"`; do json -I -f $FILE.map -e \"this.file='$(basename -- $FILE)'\" && sed 's/sourceMappingURL=\\([^ ]*\\).js.map/sourceMappingURL=\\1.mjs.map/g' $FILE > $FILE.remap && mv $FILE.remap $FILE ; done",
"build:esm:rename": "renamer --find .js --replace .mjs 'dist/esm/**/*.js*'",
},
"devDependencies": {
// ...
"json": "^9.0.6",
"npm-run-all": "^4.1.5",
"renamer": "^1.1.3"
},
}
Note also, that main vs module key in package.json is ignored by nodeJS (ref)
@phil-lgr that actually works pretty well. The one place I am stuck though is in SSR projects where we have some code that is pulling in CSS files still (for webpack). We are using ignore-styles package but that doesn't seem to work with this new system.
Node.js support for unflagged native ESM should land in Node 13.2.0 (probably next week). Here's the PR for the announcement and details.
@weswigham Are there any updates related to this? Would be greatly appreciated to have this coincide with unflagging if there are updates.
@SomethingSexy Assuming you are interested in running the ESM files in Node.js, you can explore --loader
which is still experimental but can help deal with things like CSS.
In most cases, I'd urge people to only do experimental things when experimenting, and in this case it is still true, but CSS files loaded in this context are likely merely noop and no error, right?
If that is the case, I'd recommend exploring it cautiously — see Experimental Loader hooks.
Important — Please note that experimental and/or flagged features are highly discouraged if you are publishing to NPM… etc.
It is fair to mention that browsers have looked into non-ESM imports a bit but backtracked (I believe it was JSON) which is discouraging at least for anyone planning on doing away with bundling and wanting all the mojo.
@SMotaal that is a good idea, I will look into that option for our testing. Ya in this case these files are noop.
@SomethingSexy Can you update your response with a link to this pattern once/if you have it public.
So node 13.2 has been released with .mjs support without flag: https://github.com/nodejs/node/releases/tag/v13.2.0
Can we config mjs and cjs output?
What's the TL;DR for generating .mjs
from TypeScript files, now that it's been a couple months since .mjs
support was unflagged in Node?
@phil-lgr's tsconfig.json
does most of it, though it's a bit unclear and outdated, so here's what you need to do as of Feb 2020:
"target": "esnext"
and "module": "esnext"
. This will output .js
files still, not .mjs
."type": "module"
, so Node can execute the generated .js
files with import
statements.--experimental-specifier-resolution=node
flag, so you don't have to specify the .js
extension in the import
statements.@dandv i my self use rollup to overcome that limitations it has a typescript plugin a checking and a none checking one it has options to define the output filenames.
Another option is to use tsc
for type checking only and to use babel to write proper module ouputs:
Using that approach, you'd still use tsc
with emitDeclarationOnly
to generate type definitions. But the generated JavaScript will be in proper module files and have fully qualified relative specifiers.
There's still rough edges with this since TypeScript may not properly recognize third party imports from modules but it may be a helpful workaround.
the best solution is to avoid .ts extension and go for .js extension then marking the source as module via adding type: module to a package.json near the module imports or even in the package root
then add JSDOC Annotations and turn checkJs true in tsconfig.json
@frank-dspeed I know, once upon a time when some said use ".ts"… now when it is fair to want to do ".js"… things take time I guess, and projects especially like to keep at it thinking it will be solved upstream, even if it has already.
So for folks on this thread, please research closer, TypeScript is a development tool, ".ts" is the way it was best accomplished at one point, but please don't take anyone's word for it, do your own digging. In the end, TypeScript is not going anywhere, even if you use ".js" you are still getting the best perks it has to offer today.
@demurgos Fair to note… this issue likely needs to be renamed to include "cjs" and/or "mjs", which can be achieved with something like
outputFormats
which would(string|{format: string, extension?: string})[]
.
@SMotaal assert{} was always superior + jsdoc.
I do not really care for typescript itself at present tsserver is the backbone of vs code that makes it matter not that it is a tool.
@frank-dspeed so I feel a little confused when addressing technical issues, how we sometimes mistake what is absolutely everyone's right to express (ie opinions and preferences) with what we all mutually must not dismiss on account of our own emotions on the former (ie works for everyone, not just the "selfish" me) — the whole idea of open source looking like social media sadly makes this kind of knee jerk reaction likely for anyone of us…
@SamHH @csvn — it really helps if you articulate where you disagree, please, I am inclined to venture but I do not want to be dismissive, so if the 👎 is not meant to discount the very real pain that comes from how typescript and vscode are coupled, which is really the technical point of relevance here imho, then articulating would really help everyone involved… thanks :)
@SMotaal I disagreed mostly with this statement:
TypeScript is not going anywhere, even if you use ".js" you are still getting the best perks it has to offer today.
I use Typescript in almost all projects I work on, and I always use strict: true
. Whenever I've used .js
and JSDoc
(e.g. for various smaller scripts) the experience I've had has always been vastly inferior to using .ts
. A very subjective note, but I feel readability suffers a lot when you need to add comments inside methods and next to variable assignments.
I do wish Typescript supported .mjs
so it was easier to use native ESM with Node v13 for experimenting, but I don't feel like using .js
is a great option for anything but very small projects. 🤷♂️
I don't see where we disagree honestly, yes, .mjs
from .ts
and .js
, this is fundamentally necessary for the ecosystem, but now even .cjs
needs to become factored in the whole process.
@csvn it might add context to add I've used TypeScript for ages, and I know exactly the trade-offs, why I moved to .js
files and still use TypeScript even as a compiler.
I hope you can appreciate at least the context of my comment in this thread alone, but I do not think we are arguing differently on the core ask in this op, fair?
@csvn i found out it works best when defining typescript like interfaces via @typedef then you can use short /* @type {name} / near the var so it is less verbose. you can also do @type {import('./@types....').name}
Any news on that functionality ?
@TheMrZZ you can expect this to stay around for a longer period of time 2+ Years maybe more. I was trying to do a fast fork to fix that it was not so easy it would take some weeks of fulltime work so we all are doomed.
If anyone is interested on how to configure TypeScript & Node.js 14 to work together, I just made a post on StackOverflow on how to do that here: https://stackoverflow.com/a/61305579/148072
tl;dr
tsconfig.json
:
"allowSyntheticDefaultImports": true
"lib": ["ES2020"]
"module": "ES2020"
"moduleResolution": "node"
"target": "ES2020"
package.json
:
"type": "module"
import { a } from './a.js'
(yes, .js
even if the file is named .ts
)--experimental-specifier-resolution=node
to avoid specifying file extensions)
Experimental support for ES modules just landed in Node 8.5 (changelog, PR). Since ES modules have some parsing and semantic differences, Node decided to use the
mjs
extension for ES modules (whilejs
is for the "script" target and commonjs modules).The current Typescript version (
2.5.2
) supports ES modules emission but uses thejs
extension by default. It means that to use it with Node, a post-compilation step is required to change the extension fromjs
tomjs
. This adds undesirable complexity to use native ES modules support with Node.A solution would be to add a compiler option to output
*.mjs
files when emitting ES modules.Edit (2018-03-22): The propositions below are a bit outdated. I recommend reading the issue to see the progression. See this comment for my current proposition.
Notes:
*.js
extension, many tools rely on thejs
extension.*.mts
files would compile to*.mjs
, this would be similar to*.tsx
and*.jsx
.