`ts-node` fails when ES Modules are in the dependency graph in Node.js 13+

trusktr commented 4 years ago

I basically detailed the issue in this comment: https://github.com/TypeStrong/ts-node/issues/155#issuecomment-570120923

It's a chicken-and-egg-like problem:

If we use module: 'commonjs', then if any TS files import ES Modules (indirectly in their dependency graph), then Node throws an error because CommonJS modules can not import ES Modules.
If we change to module: 'esnext', then the errors from the previous point go away, but now the .js file that is loading ts-node and calling something like require('typescript-entry-point.ts') will have a similar issue, because the call to require('typescript-entry-point.ts') will try to load an ES Module.
Finally, if we convert the .js file into an ES Module, we can not convert require('typescript-entry-point.ts') into import 'typescript-entry-point.ts' because now ES Modules don't handle .ts extensions (at least not out of the box, and it seems the old require hooks don't operate on these new identifiers)

At the moment, I'm sort of stuck, because I have dependencies in my dependency tree that are ES Modules.

The only workaround I can think of is to compile everything to .js files (ES Modules) and avoid to use ts-node.

I wonder if a combination of allowJs and ignore so that it compiles JS files would help. I haven't tried that yet.

trusktr commented 4 years ago

One way to reproduce this is to take a working application with module: 'commonjs' being passed to the register() call, make sure you're using the latest Node 13, then add an ES Module dependency (namely a .js file inside a node_modules package with type: 'module', as I believe that'll be the prime way that people will publish ES Modules from now on) on the leaf end of any branch of the dependency tree.

trusktr commented 4 years ago

Now that Node 13 is out in the wild, looks like people are starting to run into this without any flags. Eventually someone will have unpaid free time to fix it, but I think it is easier to just pre-compile everything to JS before running Node.

jkrems commented 4 years ago

If we use module: 'commonjs', then if any TS files import ES Modules (indirectly in their dependency graph), then Node throws an error because CommonJS modules can not import ES Modules.

I'm trying to understand this scenario - how did this work before? I assume existing TypeScript running via ts-node was never meant to use real ESM semantics, instead it's more similar to babel-style "using import as sugar for require". So it's de-facto CommonJS and shouldn't be loading ES modules from its dependencies (at least that should be the default behavior afaict).

Is this about existing projects breaking or about existing projects migrating from CJS-"import" to standard imports?

jkrems commented 4 years ago

Ah, I think I got it now: It's about running ts-node for module code. Which would involve compilation within the ES module loader, potentially registering the compiler after node is started up (as opposed to using a CLI flag). The short answer is that the APIs to allow that aren't yet in node, so I don't think ts-node can do anything about this for now. :(

trusktr commented 4 years ago

Not even by hooking in to the experimental load API?

jkrems commented 4 years ago

There's two independent holes in the API surface:

Being able to change the source text of something loaded from disk without breaking relative resolution. There is an open PR to address this by @GeoffreyBooth: https://github.com/nodejs/node/pull/30986
Being able to start a simple binary (ts-node) that registers a transformer and then runs user code. Right now the only way to register hooks is using a node flag (node --some-flag=ts-node main.ts). Short of forking a new node process, there's currently no API that allows to implement ts-node main.ts with native module support.

The second point is tricky because it's by design. We don't really want to allow userland code (from node's perspective) being able to switch out parts of the loading pipeline at runtime. It can lead to weird issues where the same global scope contains code expecting different module loaders and thanks to import() that's actually observable (as opposed to require).

One idea brought up by @guybedford was to have an API that can create a new global scope ("context" or "realm") with all bells and whistles, including node standard library, process global, and fresh module system. Tools like ts-node could use it to set up a new context for running the application code. But there are some downsides and questions, e.g. around garbage collection of the original "bootstrap" context and the time it would take to initialize the new context.

EntraptaJ commented 4 years ago

I created a proof of concept for using the current version of the --experimental-loader feature to transpile typescript files right before node loads it and then give node the now ESNext Javascript code with the SyntheticModule VM feature. It's fairly hacky right now but it made for a fun 6 hour session: https://github.com/KristianFJones/TS-ES-Node

Jamesernator commented 4 years ago

^ Just for above there's a new loader hook in node nightly called getSource so the following loader can be used in node nightly. Only caveats are the same as the babel typescript plugin (e.g. no const enum):

Typescript Loader

```js import fs from 'fs'; import path from 'path'; import babel from '@babel/core'; import urlUtils from 'url'; import babelPluginTransformTypescript from '@babel/plugin-transform-typescript'; function isURL(string) { try { new URL(string); return true; } catch { return false; } } const removePrivateTypes = (babel) => { return { visitor: { ClassPrivateProperty(context) { context.get('typeAnnotation')?.remove(); }, }, }; } const BABEL_OPTS = { plugins: [ babelPluginTransformTypescript, removePrivateTypes, ], parserOpts: { plugins: [ 'asyncGenerators', 'bigInt', 'classProperties', 'classPrivateProperties', 'importMeta', 'nullishCoalescingOperator', 'numericSeparator', 'objectRestSpread', 'optionalCatchBinding', 'optionalChaining', 'topLevelAwait', 'typescript', ], }, sourceMaps: 'inline', }; export async function getSource(urlString, context, getSourceDefault) { if (isURL(urlString)) { const url = new URL(urlString); if (url.pathname.endsWith('.js') && !fs.existsSync(url)) { url.pathname = url.pathname.replace(/\.js$/u, '.ts'); const contents = await fs.promises.readFile(url, 'utf8'); const { code: source } = await babel.transformAsync(contents, { ...BABEL_OPTS, sourceFileName: path.basename(urlUtils.fileURLToPath(url)), }) return { source }; } } return getSourceDefault(urlString, context, getSourceDefault); } export async function resolve(specifier, context, defaultResolve) { try { const defaultResolution = defaultResolve(specifier, context, defaultResolve); try { const url = new URL(defaultResolution.url); url.pathname = url.pathname.replace(/\.ts$/, '.js'); return { url: url.href }; } catch { return defaultResolution; } } catch { return { url: new URL(specifier, context.parentURL).href }; } } ```

GeoffreyBooth commented 4 years ago

Hi, I created the getSource hook. There's also a transformSource hook that was intended for things like transpilation. See the transpiler loader example: https://github.com/nodejs/node/blob/master/doc/api/esm.md#transpiler-loader

EntraptaJ commented 4 years ago

Look's like I know what I'm doing this weekend.

EntraptaJ commented 4 years ago

Been busy, here is a proof of concept using the transformSource hook. Supports relative imports of .ts and .tsx without requiring extensions. https://github.com/K-FOSS/TS-ESNode

EntraptaJ commented 4 years ago

Okay, made a few small changes and fixes. It's now able to handle external modules that have the TSLib import helper without requiring refactoring existing imports. The TS-ESNode loader hook can be dropped into existing projects and just work. Hopefully! I just tested it on my main Application template that uses TypeORM & TypeGraphQL with tsconfig.json set to use ESNext as target and modules and it all just works. Transpiled code is all using imports and exports with Node V14 in Module mode.

EntraptaJ commented 4 years ago

Removed the globby requirement. Module has been published on NPM https://www.npmjs.com/package/@k-foss/ts-esnode zero dependencies other then peer dependency on TypeScript. NPM reports unpacked size of 8.66 kB. Should hopefully be a drop in replacement for TS-Node for now.

blakeembrey commented 4 years ago

@KristianFJones If you're interested I'd love to land a PR in ts-node with this functionality.

EntraptaJ commented 4 years ago

I'd love to give that a shot. I'll take a crack at it sometime this weekend.

ejose19 commented 4 years ago

@KristianFJones I tried your fork and so far it works without issues, I also added --harmony-top-level-await to try along typescript 3.8 and it worked without issues. Hopefully it can be integrated into ts-node.

Btw, I saw you state it requires node 14 (nightly), but I could use 13.7 without any issues (even TLA), anything lower than that and I start getting all kinds of errors.

jkrems commented 4 years ago

Please note that TLA may look like it's working at first glance but is known to still have some sharp edges. Both the base implementation in V8 and the integration in nodejs. See: https://github.com/nodejs/node/pull/30370. So it's nice to play around with locally but things are kind of expected to break horribly at this point. It's not ready for real-world use yet.

Just in case anybody is tempted to enable that flag somewhere. :)

EntraptaJ commented 4 years ago

@ejose19 Interesting, I had been developing it with Node 13.6, didn't realize the transformSource hooks were added to 13.7. Unless you tried the one on my Account and not the new transformSource one I have on my K-FOSS Organization.

vansergen commented 4 years ago

It seems that ts-node fails to run with "module": "esnext" in the tsconfig.json and "type": "module" in the package.json with the following error:

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /Users/user/test-project/index.ts
    at Loader.defaultGetFormat [as _getFormat] (internal/modules/esm/get_format.js:71:15)
    at Loader.resolve (internal/modules/esm/loader.js:98:42)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at Loader.getModuleJob (internal/modules/esm/loader.js:188:29)
    at Loader.import (internal/modules/esm/loader.js:163:17)

EntraptaJ commented 4 years ago

Well, looks like @K-FOSS/TS-ESNode does work with Node 13.7. That's amazing. I'll get to work on porting this directly into ts-node hopefully sometime this weekend. Work has been busy, not enough time for personal coding.

ai commented 4 years ago

@KristianFJones if you can describe how we can help you here, I can ask my friends to make pull request

droganov commented 4 years ago

can it be related to https://github.com/nodejs/node/issues/32103 ?

EntraptaJ commented 4 years ago

@KristianFJones if you can describe how we can help you here, I can ask my friends to make pull request

My hesitation with making a port/adaptation of TS-ESNode for ts-node is I'm not sure if the core of TS-ESNode truly belongs in TS-Node without a full refactor of the existing code base. I could easily just copy over my TS-ESNode work into ts-node/node-hooks and have it work node --loader ts-node/node-hooks but I'm not sure if it makes sense to have almost a fully different approach to loading TypeScript into Node.JS in an existing library.

EntraptaJ commented 4 years ago

can it be related to nodejs/node#32103 ?

No, I've written a full TypeScript discovery system to provide node style resolution within ESModule mode using my loader hook to automatically discover .ts,.tsx,.js,.jsx files when no existing file is provided to the hook.

https://github.com/K-FOSS/TS-ESNode/blob/next/src/findFiles.ts

cspotcode commented 4 years ago

This is released as an experimental feature in ts-node v8.10.0. Please test and share your feedback in #1007.

You will probably want to enable "transpileOnly" in your tsconfig for better performance. However, we do support typechecking if you want that.

natesire commented 4 years ago

Yes, this issue happened right after I added "type": "module" to the package.json in an attempt to not create the .mjs files. I am on node 14

webia1 commented 4 years ago

Same here:

tsconfig.json -> compilerOptions:

"target": "es2017",
"module": "esnext"

package.json:

"type": "module"

Reason for the entries above:

Top-level 'await' expressions are only allowed when the 'module' option is set to 'esnext' or 'system', and the 'target' option is set to 'es2017' or higher. ts(1378)

meabed commented 4 years ago

yea please - this need a quick fix - no one can use top-level-await with ts-node :(

EntraptaJ commented 4 years ago

yea please - this need a quick fix - no one can use top-level-await with ts-node :(

But you can with @K-FOSS/TS-ESNode

meabed commented 4 years ago

yea please - this need a quick fix - no one can use top-level-await with ts-node :(

But you can with @K-FOSS/TS-ESNode

sure but you would have bunch of other un-resolved issues as well :( It would be great if this top-level-await supported properly on ts-node as its really code changer for so many use-cases.

cspotcode commented 4 years ago

@meabed have you tried using ts-node's built-in ESM support? #1007 It should allow you to use top-level await. If it is not working, please file a bug with a reproduction so we can make a fix.

meabed commented 4 years ago

@meabed have you tried using ts-node's built-in ESM support? #1007 It should allow you to use top-level await. If it is not working, please file a bug with a reproduction so we can make a fix.

Thanks @cspotcode - its not working - I will make sandbox - to simulate the errors, I get module not found.

Ofcourse I don't know whats the effort to get this integrated and tested properly - I appreciate the your effort and the community effort to get this resolved.

I will share sandbox to reproduce and test in another comment shortly. Thanks alot!

ejose19 commented 4 years ago

@meabed are you sure it's not working? I just tried and and it works correctly (must use node v14.3.0+ with tla flag)

tsconfig.json

{
  "compilerOptions": {
    "target": "ES2017",
    "module": "ESNext"
  }
}

package.json

{
  "scripts": {
    "start": "node --loader ts-node/esm.mjs --experimental-top-level-await index.ts"
  },
  "dependencies": {
    "@types/node": "^14.0.13",
    "ts-node": "^8.10.2",
    "typescript": "^3.9.5"
  },
  "type": "module"
}

index.ts

import { promisify } from "util";

const sleep = promisify(setTimeout);

console.log("start");
await sleep(2000);
console.log("finish");

cspotcode commented 4 years ago

@ejose19 Thank you for sharing an example.

@meabed given the example, is ts-node working for you? If not, have you had any success creating a reproduction?

meabed commented 4 years ago

Hi @cspotcode Thanks alot for the followup :) the implementation above works :)

Thanks a ton @ejose19 for the example! it works perfectly :) I have another issue when i import named exports from another .ts - I get "ERR_MODULE_NOT_FOUND" error.

// for example file test-export.ts 
const car = {name:"BMW"};
export {car};
// then in index.ts 
import {car} from './test-export'; // this gives ERR_MODULE_NOT_FOUND
console.log(car);

cspotcode commented 4 years ago

@meabed This is a problem with your code. The solution is explained in the "Usage" section of the documentation here: #1007. Please read it fully, because it also explains some other details you need to be aware of.

meabed commented 4 years ago

Thanks a lot @cspotcode i am looking at that now and I’ll update my comment to reflect the solution :) thanks for the hint and support 👍

meabed commented 4 years ago

Thanks @cspotcode - it works perfectly fine :) Thanks alot! the missing part was importing the file with .js extension

import {car} from './test-export.js';

thats awesome it works :) a follow up question - do you think it would be doable without the extension import soon 👍 just trying to see should we all imports to .js to use the top level await or we hold a bit if its coming soon. Best regards.

cspotcode commented 4 years ago

@meabed that is also explained in the "Usage" section of #1007 and in node's ESM documentation. Remember that a lot of things are controlled by node and TypeScript. At the end of the day, we're using TypeScript to convert your code into JS that's run by node, and we need to remain compatible with tsc && node. A great way to answer these questions for yourself is to try using tsc && node and see what is and is not allowed.

katywings commented 4 years ago

I have an issue analog to the reproduction https://github.com/TypeStrong/ts-node/issues/935#issuecomment-570124674. My dependency microbundle uses autoprefixer which uses colorette and colorette switched to esm. Now I get

require() of ./node_modules/colorette/index.js from ./node_modules/autoprefixer/lib/autoprefixer.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.

Do I understand it correctly that I only can fix this by making my own package a module as well, at the moment?

cspotcode commented 4 years ago

@katywings no, that sounds like a bug in colorette. Based on the error, a non-ts file is requiring another non-ts file, so ts-node is not involved at all.

katywings commented 4 years ago

@cspotcode Thanks for your feedback :). microbundle which uses autoprefixer which uses colorette works with plain node. The error only happens if ts-node is involved, so to me it looks like ts-node obviously has to be "involved" ^^.

This is the trace: at Module._extensions..js (internal/modules/cjs/loader.js:1217:13) at Object.require.extensions. [as .js] (XYZ/node_modules/ts-node/src/index.ts:851:44) at Module.load (internal/modules/cjs/loader.js:1050:32) at Function.Module._load (internal/modules/cjs/loader.js:938:14) at Module.require (internal/modules/cjs/loader.js:1090:19) at require (internal/modules/cjs/helpers.js:75:18) at Object. (XYZ/node_modules/autoprefixer/lib/autoprefixer.js:5:17) at Module._compile (internal/modules/cjs/loader.js:1201:30) at Module._extensions..js (internal/modules/cjs/loader.js:1221:10) at Object.require.extensions. [as .js] (XYZ/node_modules/ts-node/src/index.ts:851:44)

cspotcode commented 4 years ago

@katywings Thanks for the details. Do you have allowJs turned on? What's the node version? Can you send a minimal reproducible example?

Colorette declares that require() should be loading index.cjs, not index.js. https://unpkg.com/browse/colorette@1.2.1/package.json

But the stack trace shows that it's trying to load index.js.

That leads me to suspect a node bug that manifests when a custom require.extensions['.js'] handler is installed. But without a reproduction, there are too many unknowns: I don't know the versions of libraries or of node and I don't know the ts-node nor ts configurations.

I think your issue is subtley different than the ones described in this ticket, because ESM should not come into play. Colorette is telling node that it can be loaded as CommonJS but somehow that's being ignored.

Feel free to file this as a separate ticket if you want.

katywings commented 4 years ago

@cspotcode After your mentioned details and the info about the subtle difference I worked out a minimal reproduction case and with that I figured out that the error happens because of some side effect with tsconfig-paths and a baseUrl of ./node_modules, aiaiai :D.

cspotcode commented 4 years ago

@katywings Thanks for tracking this down and making a ticket. Sure enough, tsconfig-paths needs to hook into node's resolver and I guess it hasn't been updated to account for node's newer resolver behaviors.

https://github.com/dividab/tsconfig-paths/blob/master/src/register.ts#L73-L96

cspotcode commented 4 years ago

I'm closing this issue because the feature has been implemented, and feedback is tracked by #1007.

BuistvoPloti commented 3 years ago

same issues occurring right now

mysterycommand commented 3 years ago

If anyone is coming to this issue like I did I was able to get TS working inside a project with "type": "module" through nodemon by adding a nodemon.json with this in it:

{
  "execMap": {
    "ts": "node --loader ts-node/esm"
  }
}

Essentially replacing the default "execMap" of "ts": "ts-node" … in case it matters my tsconfig.json like:

{
  "compilerOptions": {
    "module": "ESNext",
    "target": "ESNext",
    "moduleResolution": "node",
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "isolatedModules": true,
    "noEmit": true,
    "strict": true,
    "lib": [
      "es2020"
    ]
  }
}

With that my "dev" script in package.json is just "nodemon src/index.ts" and it all seems to work!

umershaekoor commented 3 years ago

still having the same issue. @mysterycommand's solution doesnt work for me either. trying the same thing as the original author of this thread. Attempting to import a library with es6 modules in a ts project and trying to run it with ts-node <path-to-entry-point.ts> my tsconfig.json is as follows


{
  "compilerOptions": {
    "target": "es2017",
    "module": "commonjs",
    "lib": [
      "es2020"
    ],

    "allowJs": true,
    "declaration": true,
    "declarationMap": true,
    "sourceMap": true,
    "outDir": "./dist",
    "composite": true,
    "strict": true,
    "moduleResolution": "node",
    "rootDirs": [
      "."
    ],

    "esModuleInterop": true,

    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true
  },
  "references": [],
  "watchOptions": {
    "fallbackPolling": "dynamicPriority",
    "excludeDirectories": [
      "**/node_modules",
      "build"
    ]
  }
}```

cspotcode commented 3 years ago

@BLitzMe

trying to run it with ts-node

That looks like the problem. See #1007

TypeStrong / ts-node

`ts-node` fails when ES Modules are in the dependency graph in Node.js 13+ #935