Closed trusktr closed 4 years ago
One way to reproduce this is to take a working application with module: 'commonjs'
being passed to the register()
call, make sure you're using the latest Node 13, then add an ES Module dependency (namely a .js
file inside a node_modules package with type: 'module'
, as I believe that'll be the prime way that people will publish ES Modules from now on) on the leaf end of any branch of the dependency tree.
related:
Now that Node 13 is out in the wild, looks like people are starting to run into this without any flags. Eventually someone will have unpaid free time to fix it, but I think it is easier to just pre-compile everything to JS before running Node.
If we use module: 'commonjs', then if any TS files import ES Modules (indirectly in their dependency graph), then Node throws an error because CommonJS modules can not import ES Modules.
I'm trying to understand this scenario - how did this work before? I assume existing TypeScript running via ts-node
was never meant to use real ESM semantics, instead it's more similar to babel-style "using import as sugar for require". So it's de-facto CommonJS and shouldn't be loading ES modules from its dependencies (at least that should be the default behavior afaict).
Is this about existing projects breaking or about existing projects migrating from CJS-"import" to standard imports?
Ah, I think I got it now: It's about running ts-node
for module code. Which would involve compilation within the ES module loader, potentially registering the compiler after node is started up (as opposed to using a CLI flag). The short answer is that the APIs to allow that aren't yet in node, so I don't think ts-node
can do anything about this for now. :(
Not even by hooking in to the experimental load API?
There's two independent holes in the API surface:
ts-node
) that registers a transformer and then runs user code. Right now the only way to register hooks is using a node flag (node --some-flag=ts-node main.ts
). Short of forking a new node process, there's currently no API that allows to implement ts-node main.ts
with native module support.The second point is tricky because it's by design. We don't really want to allow userland code (from node's perspective) being able to switch out parts of the loading pipeline at runtime. It can lead to weird issues where the same global scope contains code expecting different module loaders and thanks to import()
that's actually observable (as opposed to require
).
One idea brought up by @guybedford was to have an API that can create a new global scope ("context" or "realm") with all bells and whistles, including node standard library, process
global, and fresh module system. Tools like ts-node
could use it to set up a new context for running the application code. But there are some downsides and questions, e.g. around garbage collection of the original "bootstrap" context and the time it would take to initialize the new context.
I created a proof of concept for using the current version of the --experimental-loader feature to transpile typescript files right before node loads it and then give node the now ESNext Javascript code with the SyntheticModule VM feature. It's fairly hacky right now but it made for a fun 6 hour session: https://github.com/KristianFJones/TS-ES-Node
^ Just for above there's a new loader hook in node nightly called getSource
so the following loader can be used in node nightly. Only caveats are the same as the babel typescript plugin (e.g. no const enum
):
Hi, I created the getSource
hook. There's also a transformSource
hook that was intended for things like transpilation. See the transpiler loader example: https://github.com/nodejs/node/blob/master/doc/api/esm.md#transpiler-loader
Look's like I know what I'm doing this weekend.
Been busy, here is a proof of concept using the transformSource
hook. Supports relative imports of .ts
and .tsx
without requiring extensions. https://github.com/K-FOSS/TS-ESNode
Okay, made a few small changes and fixes. It's now able to handle external modules that have the TSLib import helper without requiring refactoring existing imports. The TS-ESNode
loader hook can be dropped into existing projects and just work. Hopefully! I just tested it on my main Application template that uses TypeORM & TypeGraphQL with tsconfig.json
set to use ESNext as target and modules and it all just works. Transpiled code is all using imports and exports with Node V14 in Module mode.
Removed the globby
requirement. Module has been published on NPM https://www.npmjs.com/package/@k-foss/ts-esnode zero dependencies other then peer dependency on TypeScript. NPM reports unpacked size of 8.66 kB. Should hopefully be a drop in replacement for TS-Node
for now.
@KristianFJones If you're interested I'd love to land a PR in ts-node
with this functionality.
I'd love to give that a shot. I'll take a crack at it sometime this weekend.
@KristianFJones I tried your fork and so far it works without issues, I also added --harmony-top-level-await to try along typescript 3.8 and it worked without issues. Hopefully it can be integrated into ts-node.
Btw, I saw you state it requires node 14 (nightly), but I could use 13.7 without any issues (even TLA), anything lower than that and I start getting all kinds of errors.
Please note that TLA may look like it's working at first glance but is known to still have some sharp edges. Both the base implementation in V8 and the integration in nodejs. See: https://github.com/nodejs/node/pull/30370. So it's nice to play around with locally but things are kind of expected to break horribly at this point. It's not ready for real-world use yet.
Just in case anybody is tempted to enable that flag somewhere. :)
@ejose19 Interesting, I had been developing it with Node 13.6, didn't realize the transformSource hooks were added to 13.7. Unless you tried the one on my Account and not the new transformSource one I have on my K-FOSS
Organization.
It seems that ts-node
fails to run with "module": "esnext"
in the tsconfig.json
and "type": "module"
in the package.json
with the following error:
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /Users/user/test-project/index.ts
at Loader.defaultGetFormat [as _getFormat] (internal/modules/esm/get_format.js:71:15)
at Loader.resolve (internal/modules/esm/loader.js:98:42)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at Loader.getModuleJob (internal/modules/esm/loader.js:188:29)
at Loader.import (internal/modules/esm/loader.js:163:17)
Well, looks like @K-FOSS/TS-ESNode
does work with Node 13.7. That's amazing. I'll get to work on porting this directly into ts-node
hopefully sometime this weekend. Work has been busy, not enough time for personal coding.
@KristianFJones if you can describe how we can help you here, I can ask my friends to make pull request
can it be related to https://github.com/nodejs/node/issues/32103 ?
@KristianFJones if you can describe how we can help you here, I can ask my friends to make pull request
My hesitation with making a port/adaptation of TS-ESNode for ts-node
is I'm not sure if the core of TS-ESNode
truly belongs in TS-Node
without a full refactor of the existing code base. I could easily just copy over my TS-ESNode work into ts-node/node-hooks
and have it work node --loader ts-node/node-hooks
but I'm not sure if it makes sense to have almost a fully different approach to loading TypeScript into Node.JS in an existing library.
can it be related to nodejs/node#32103 ?
No, I've written a full TypeScript discovery system to provide node style resolution within ESModule mode using my loader hook to automatically discover .ts,.tsx,.js,.jsx files when no existing file is provided to the hook.
https://github.com/K-FOSS/TS-ESNode/blob/next/src/findFiles.ts
This is released as an experimental feature in ts-node v8.10.0. Please test and share your feedback in #1007.
You will probably want to enable "transpileOnly" in your tsconfig for better performance. However, we do support typechecking if you want that.
Yes, this issue happened right after I added "type": "module" to the package.json in an attempt to not create the .mjs files. I am on node 14
Same here:
tsconfig.json -> compilerOptions:
"target": "es2017",
"module": "esnext"
package.json:
"type": "module"
Reason for the entries above:
Top-level 'await' expressions are only allowed when the 'module' option is set to 'esnext' or 'system', and the 'target' option is set to 'es2017' or higher. ts(1378)
yea please - this need a quick fix - no one can use top-level-await with ts-node :(
yea please - this need a quick fix - no one can use top-level-await with ts-node :(
But you can with @K-FOSS/TS-ESNode
yea please - this need a quick fix - no one can use top-level-await with ts-node :(
But you can with @K-FOSS/TS-ESNode
sure but you would have bunch of other un-resolved issues as well :( It would be great if this top-level-await supported properly on ts-node as its really code changer for so many use-cases.
@meabed have you tried using ts-node's built-in ESM support? #1007 It should allow you to use top-level await. If it is not working, please file a bug with a reproduction so we can make a fix.
@meabed have you tried using ts-node's built-in ESM support? #1007 It should allow you to use top-level await. If it is not working, please file a bug with a reproduction so we can make a fix.
Thanks @cspotcode - its not working - I will make sandbox - to simulate the errors, I get module not found.
Ofcourse I don't know whats the effort to get this integrated and tested properly - I appreciate the your effort and the community effort to get this resolved.
I will share sandbox to reproduce and test in another comment shortly. Thanks alot!
@meabed are you sure it's not working? I just tried and and it works correctly (must use node v14.3.0+ with tla flag)
tsconfig.json
{
"compilerOptions": {
"target": "ES2017",
"module": "ESNext"
}
}
package.json
{
"scripts": {
"start": "node --loader ts-node/esm.mjs --experimental-top-level-await index.ts"
},
"dependencies": {
"@types/node": "^14.0.13",
"ts-node": "^8.10.2",
"typescript": "^3.9.5"
},
"type": "module"
}
index.ts
import { promisify } from "util";
const sleep = promisify(setTimeout);
console.log("start");
await sleep(2000);
console.log("finish");
@ejose19 Thank you for sharing an example.
@meabed given the example, is ts-node working for you? If not, have you had any success creating a reproduction?
Hi @cspotcode Thanks alot for the followup :) the implementation above works :)
Thanks a ton @ejose19 for the example! it works perfectly :) I have another issue when i import named exports from another .ts - I get "ERR_MODULE_NOT_FOUND" error.
// for example file test-export.ts
const car = {name:"BMW"};
export {car};
// then in index.ts
import {car} from './test-export'; // this gives ERR_MODULE_NOT_FOUND
console.log(car);
@meabed This is a problem with your code. The solution is explained in the "Usage" section of the documentation here: #1007. Please read it fully, because it also explains some other details you need to be aware of.
Thanks a lot @cspotcode i am looking at that now and I’ll update my comment to reflect the solution :) thanks for the hint and support 👍
Thanks @cspotcode - it works perfectly fine :) Thanks alot! the missing part was importing the file with .js extension
import {car} from './test-export.js';
thats awesome it works :) a follow up question - do you think it would be doable without the extension import soon 👍 just trying to see should we all imports to .js to use the top level await or we hold a bit if its coming soon. Best regards.
@meabed that is also explained in the "Usage" section of #1007 and in node's ESM documentation. Remember that a lot of things are controlled by node and TypeScript. At the end of the day, we're using TypeScript to convert your code into JS that's run by node, and we need to remain compatible with tsc && node
. A great way to answer these questions for yourself is to try using tsc && node
and see what is and is not allowed.
I have an issue analog to the reproduction https://github.com/TypeStrong/ts-node/issues/935#issuecomment-570124674. My dependency microbundle uses autoprefixer which uses colorette and colorette switched to esm. Now I get
require() of ./node_modules/colorette/index.js from ./node_modules/autoprefixer/lib/autoprefixer.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Do I understand it correctly that I only can fix this by making my own package a module as well, at the moment?
@katywings no, that sounds like a bug in colorette. Based on the error, a non-ts file is requiring another non-ts file, so ts-node is not involved at all.
@cspotcode Thanks for your feedback :). microbundle which uses autoprefixer which uses colorette works with plain node. The error only happens if ts-node is involved, so to me it looks like ts-node obviously has to be "involved" ^^.
This is the trace:
at Module._extensions..js (internal/modules/cjs/loader.js:1217:13)
at Object.require.extensions.
@katywings Thanks for the details. Do you have allowJs turned on? What's the node version? Can you send a minimal reproducible example?
Colorette declares that require()
should be loading index.cjs
, not index.js
. https://unpkg.com/browse/colorette@1.2.1/package.json
But the stack trace shows that it's trying to load index.js
.
That leads me to suspect a node bug that manifests when a custom require.extensions['.js']
handler is installed. But without a reproduction, there are too many unknowns: I don't know the versions of libraries or of node and I don't know the ts-node nor ts configurations.
I think your issue is subtley different than the ones described in this ticket, because ESM should not come into play. Colorette is telling node that it can be loaded as CommonJS but somehow that's being ignored.
Feel free to file this as a separate ticket if you want.
@cspotcode After your mentioned details and the info about the subtle difference I worked out a minimal reproduction case and with that I figured out that the error happens because of some side effect with tsconfig-paths and a baseUrl of ./node_modules
, aiaiai :D.
@katywings Thanks for tracking this down and making a ticket. Sure enough, tsconfig-paths needs to hook into node's resolver and I guess it hasn't been updated to account for node's newer resolver behaviors.
https://github.com/dividab/tsconfig-paths/blob/master/src/register.ts#L73-L96
I'm closing this issue because the feature has been implemented, and feedback is tracked by #1007.
same issues occurring right now
If anyone is coming to this issue like I did I was able to get TS working inside a project with "type": "module"
through nodemon
by adding a nodemon.json
with this in it:
{
"execMap": {
"ts": "node --loader ts-node/esm"
}
}
Essentially replacing the default "execMap"
of "ts": "ts-node"
… in case it matters my tsconfig.json
like:
{
"compilerOptions": {
"module": "ESNext",
"target": "ESNext",
"moduleResolution": "node",
"esModuleInterop": true,
"allowSyntheticDefaultImports": true,
"isolatedModules": true,
"noEmit": true,
"strict": true,
"lib": [
"es2020"
]
}
}
With that my "dev"
script in package.json
is just "nodemon src/index.ts"
and it all seems to work!
still having the same issue. @mysterycommand's solution doesnt work for me either.
trying the same thing as the original author of this thread. Attempting to import a library with es6 modules in a ts project and trying to run it with ts-node <path-to-entry-point.ts>
my tsconfig.json is as follows
{
"compilerOptions": {
"target": "es2017",
"module": "commonjs",
"lib": [
"es2020"
],
"allowJs": true,
"declaration": true,
"declarationMap": true,
"sourceMap": true,
"outDir": "./dist",
"composite": true,
"strict": true,
"moduleResolution": "node",
"rootDirs": [
"."
],
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true
},
"references": [],
"watchOptions": {
"fallbackPolling": "dynamicPriority",
"excludeDirectories": [
"**/node_modules",
"build"
]
}
}```
@BLitzMe
trying to run it with ts-node
That looks like the problem. See #1007
I basically detailed the issue in this comment: https://github.com/TypeStrong/ts-node/issues/155#issuecomment-570120923
It's a chicken-and-egg-like problem:
module: 'commonjs'
, then if any TS files import ES Modules (indirectly in their dependency graph), then Node throws an error because CommonJS modules can not import ES Modules.module: 'esnext'
, then the errors from the previous point go away, but now the.js
file that is loadingts-node
and calling something likerequire('typescript-entry-point.ts')
will have a similar issue, because the call torequire('typescript-entry-point.ts')
will try to load an ES Module..js
file into an ES Module, we can not convertrequire('typescript-entry-point.ts')
intoimport 'typescript-entry-point.ts'
because now ES Modules don't handle.ts
extensions (at least not out of the box, and it seems the old require hooks don't operate on these new identifiers)At the moment, I'm sort of stuck, because I have dependencies in my dependency tree that are ES Modules.
The only workaround I can think of is to compile everything to
.js
files (ES Modules) and avoid to use ts-node.I wonder if a combination of
allowJs
andignore
so that it compiles JS files would help. I haven't tried that yet.