cspotcode commented 4 years ago

Please use this ticket to provide feedback on our native ESM support. Your involvement is greatly appreciated to ensure the feature works on real-world projects.

Experimental warning

Node's loader hooks are EXPERIMENTAL and subject to change. ts-node's ESM support is as stable as it can be, but it relies on APIs which node can and will break in new versions of node.

When node breaks their APIs, it breaks loaders using their APIs. You have been warned!

Third-party docs: "Guide: ES Modules in NodeJS"

Someone has been maintaining a great reference document explaining how to use ts-node's ESM loader.

Guide: ES Modules in NodeJS

First-party docs

Our website explains the basics:

CommonJS vs native ECMAScript modules Options: esm

Usage

Requirements

Set "module": "ESNext" or "ES2015" so that TypeScript emits import/export syntax.
Set "type": "module" in your package.json, which is required to tell node that .js files are ESM instead of CommonJS. To be compatible with editors, the compiler, and the TypeScript ecosystem, we cannot name our source files .mts nor .mjs.
Include file extensions in your import statements, or pass --experimental-specifier-resolution=node Idiomatic TypeScript should import foo.ts as import 'foo.js'; TypeScript understands this.
- The language service accepts configuration to include the file extension in automatically-written imports. In VSCode:

Invocation

ts-node-esm ./my-script.ts

ts-node --esm ./my-script.ts

# If you add "esm": true to your tsconfig, you can omit the CLI flag
ts-node ./my-script.ts

# If you must invoke node directly, pass --loader
node --loader ts-node/esm ./my-script.ts

# To force the use of a specific tsconfig.json, use the TS_NODE_PROJECT environment variable
TS_NODE_PROJECT="path/to/tsconfig.json" node --loader ts-node/esm ./my-script.ts

# To install the loader into a node-based CLI tool, use NODE_OPTIONS
NODE_OPTIONS='--loader ts-node/esm' greeter --config ./greeter.config.ts sayhello

ts-node-esm / --esm / "esm": true work by spawning a subprocess and passing it the --loader flag.

Configuration

When running ts-node --esm, ts-node-esm, or ts-node all CLI flags and configuration are parsed as normal. However, when passing --loader ts-node/esm, the following limitations apply:

tsconfig.json is parsed.
CLI flags are not parsed.
Environment variables are parsed.
ts-node must be installed locally, not globally. npm install ts-node or yarn add ts-node.

tsconfig will be resolved relative to process.cwd() or to TS_NODE_PROJECT. Specify ts-node options in your tsconfig file. For details, see our docs.

Use TS_NODE_PROJECT to tell ts-node to use a specific tsconfig, and put all ts-node options into this config file.

Versioning

As long as node's APIs are experimental, all changes to ESM support in ts-node, including breaking changes, will be released as minor or patch versions, NOT major versions. This conforms to semantic versioning's philosophy for version numbers lower than 1.0. Stable features will continue to be versioned as normal.

node's API change: v16.12.0, v17.0.0

Node made a breaking change in their ESM API in version 17, backported to 16.12.0. It may also be backported to 14 and 12. This is the change: nodejs/node#37468

ts-node automatically supports both APIs, thanks to #1457. This relies on hard-coded version number checks. If/when this is backported to node 14 and 12, we will publish a new version of ts-node with the appropriate version number checks. Be sure you are always using the latest version of ts-node to avoid problems.

Note: things below this line may be out-of-date or inaccurate. These notes were used during initial implementation, but have not been updated since

Pending development work

[ ] Make resolution lookup use our fs caches
[ ] Create esm-script.mjs to do --script-mode?
- Can read process.argv for config resolution?
[x] Implement require('ts-node').esmImport(module, 'import-path')
- use https://npm.im/tsimportlib
[x] Throw error when CJS attempts to require ESM, matching node's behavior for .js
- See below: "Changes to existing functionality" > "require() hook"

The proposal

Below is the official proposal, explaining our implementation in detail.

I am asking node's modules team questions here: https://github.com/nodejs/modules/issues/351

I was reading the threads about ESM support in ts-node, e.g. #935.

The @K-FOSS/TS-ESNode implementation is unfortunately incomplete; it does not attempt to typecheck. (it uses transpileModule)

So I did some research. Below is a proposal for ESM support in ts-node, describing the required behavior in detail.

This doesn't feel like an urgent feature to me, but I like having an official proposal we can work on.

Usage

node --loader ts-node/esm ./entrypoint.ts

Cannot be invoked as ts-node because it requires node flags; hooks cannot be enabled at runtime. This is unavoidable.

For simplicity, --require ts-node/register can be eliminated, because ts-node/esm automatically does that.

Alternatively, we publish an experimental ts-node-esm entry-point which invokes a node subprocess.

Don't forget allowJs! Affects the treatment of .js files. (Not .mjs nor .cjs because the TS language service won't look at them)

ESM hooks

Must implement ESM hooks to resolve extensionless imports to .ts files, resolve .js to .ts, classify .ts(x) and .jsx files as CJS or MJS, and compile .ts(x) and .jsx files.

`resolve()` hook:

Match additional file extensions: .ts, .tsx, .jsx.

Resolve .ts, .tsx, and .jsx if the import specifier says .js. Obey preferTsExts when doing this.

_

[Good idea?] Always ask default resolver first. If it finds something, we should not interfere.

--experimental-specifier-resolution=node does not obey require.extensions, unfortunately, so we can't use that.

`getFormat` hook:

If the resolved file is .ts, .tsx, or .jsx, behave as if extension was .js: use node's package.json discovery behavior to figure out if ESM or CJS.

This can be accomplished by appending .js to the URL path and delegating to built-in getFormat hook.

`transformSource` hook:

Same as today's code transformer. Relies on projects to be configured correctly for import/export emit.

Changes to existing functionality

`require()` hook

Use same getFormat logic to determine if node will treat file as CJS or ESM.
NOTE node already detects and throws some errors on its own. But if require.resolve points to a .ts file, we need to make the determination.
If ESM, throw the same error as NodeJS ("cannot load ESM via require()")
- copying logic from node's source here

`require()` code transform

Must somehow allow import() calls.
Force consumers to use require('ts-node').esmImport(module, 'import-path')?

`ts-node` bin entry-point

ts-node CLI does NOT need to support import()ing ESM.

WHY? Because ESM hooks are an experimental feature which must be enabled via node CLI flag.

Thus we will be loaded via --require, and Node is responsible for loading the entry-point, either triggering our hook or our require.extensions.

Allow `import()` in CJS

If "module": "commonjs", compiler transforms import() into __importStar

No way to change this without a custom transformer, which IMO is too much complexity at this time.

Users should run their code as ESM.

If they can't do that, we can recommend the following workaround:

// This is in a CommonJS file:
const dynamicallyImportedEsmModule = await require('ts-node').importESM('./specifier-of-esm-module', module);

Emit considerations

NOTE we have not implemented the following, although initially I thought we might. Instead, we assume tsconfig is configured for either ESM or CJS as needed

We could intelligently emit both "module": "esnext" and "module": "commonjs" depending on the classification of a file.

In transpile-only mode this is simple. Call transpileModule with different options.

When typechecking, we can pull SourceFile ASTs from the language service / incremental compiler.

We'll need a second compiler, one for each emit format. Or we can hack it by using transpileModule for all ESM output. transpileModule is incompatible with certain kinds of TS code, (can't do const enums) but it might work for a first-pass implementation.

cspotcode commented 4 years ago

TODO: turns out, users can tell the language service to include the .js file extension with automatically-written imports. So we do not need to automatically add them, though we do need to check if a .js import might point to a .ts or .tsx file.

The option is passed to the language service in a ts.UserPreferences object. https://discordapp.com/channels/508357248330760243/640177429775777792/703301413337432114

cspotcode commented 4 years ago

I was trying to figure out if ts-node needs to automatically switch the "module" option between CommonJS and ESNext depending if we need to emit CommonJS or ESM. I concluded we do not want to do this. Here's an explanation anyway, in case I am proven wrong.

Today, ts-node respects the tsconfig's module option. Users are required to set it appropriately. If the user incorrectly sets module to ESNext and then tries to require() a TS file, they get an error because the emitted code has import statements.

Alternatively, we can automatically override the module option to be CommonJS when emitting for require() and ESNext when emitting for ESM. This allows a single tsconfig to be used for both ESM and CommonJS.

After thinking about this, it doesn't make sense. Users will choose either ESM or CommonJS via their package.json file. They won't do a mix of both. Also, this would get pretty messy since we'd be doing something that doesn't match tsc's output.

Nevertheless, if we wanted to implement this:

If the module option is already correct, we can use the languageService's getEmitOutput() like we do today. If not, we can grab a reference to the SourceFile and transform it using the same technique as transpileModule's implementation. This allow custom emit while avoiding an expensive parse.

TypeScript has an internal sourceFileAffectingCompilerOptions array. If any of those options differ, a SourceFile cannot be reused. However, some are only relevant if you care about diagnostics. For swapping out the module flag, I think SourceFile can always be reused.

cspotcode commented 4 years ago

We have released an experimental implementation of this in v8.10.1. Please test and share your feedback here.

chpeters commented 4 years ago

Thanks @cspotcode for the release! Everything seems be working minus one snafu. Importing named exports don't seem to be working, but this may be a Node module quirk. For example in index.ts:

import { graphql } from 'graphql';

will cause a syntax error of:

SyntaxError: The requested module 'graphql' does not provide an export named 'graphql'

but this can be solved by using destructuring:

import Graphql from 'graphql';
const { graphql } = Graphql;

Any way to support importing named exports in ts files?

blakeembrey commented 4 years ago

@chpeters I'd guess that would be because graphql is actually CommonJS and not an ES module. You can read more about it here: https://nodejs.org/api/esm.html#esm_interoperability_with_commonjs. Unfortunately it'll probably be messy for a while with TypeScript since the imports syntax is overloaded to represent both CommonJS and native ES modules.

NeilujD commented 4 years ago

Using mocha and TypeScript with ES modules I am facing an issue and I don't quite understand it.

Running this cmd as my test cmd :

node --experimental-modules --loader ts-node/esm.mjs ./node_modules/mocha/bin/mocha --extension ts

I get this error :

import './unit/authentication.js';
^^^^^^

SyntaxError: Cannot use import statement outside a module

What did I do wrong ?

PS: I have my tsconfig.json module attribute set to "ES2015", my package.json type attribute to "module", ts-node installed locally

cspotcode commented 4 years ago

Please send me a minimal reproduction and I'll be able to tell you.

On Fri, May 8, 2020, 12:01 PM Julien Collard notifications@github.com wrote:

Using mocha and TypeScript with ES modules I am facing an issue and I don't quite understand it.

Running this cmd as my test cmd :

node --experimental-modules --loader ts-node/esm.mjs ./node_modules/mocha/bin/mocha --extension ts

I get this error :

import './unit/authentication.js';^^^^^^ SyntaxError: Cannot use import statement outside a module

What did I do wrong ?

PS: I have my tsconfig.json module attribute set to "ES2015", my package.json type attribute to "module", ts-node installed locally

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/TypeStrong/ts-node/issues/1007#issuecomment-625886264, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC35OCOQXHWRG4OZI2UC63RQQUFZANCNFSM4MGJCWPA .

NeilujD commented 4 years ago

This is my project architecture :

src
  |_index.ts
test
  |_tests.ts
  |_unit
      |_authentication.ts
package.json
tsconfig.json

My package.json :

{
  "name": "my-project",
  "version": "1.0.0",
  "description": "My project",
  "main": "lib/index",
  "type": "module",
  "files": [
    "lib/**/*"
  ],
  "directories": {
    "test": "test"
  },
  "scripts": {
    "build": "tsc",
    "test": "node --experimental-modules --loader ts-node/esm.mjs ./node_modules/mocha/bin/mocha --extension ts"
  },
  "devDependencies": {
    "@types/chai": "^4.2.11",
    "@types/mocha": "^7.0.2",
    "@types/node": "^13.13.5",
    "chai": "^4.2.0",
    "mocha": "^7.1.2",
    "ts-node": "^8.10.1",
    "typescript": "^3.8.3"
  }
}

My tsconfig.json :

{
  "compilerOptions": {
    "target": "ES2015", 
    "module": "ES2015", 
    "lib": ["es6"], 
    "declaration": true,
    "outDir": "lib",
    "rootDir": "src",
    "strict": true, 
    "noImplicitAny": true,   
    "moduleResolution": "node",   
    "esModuleInterop": true,  
    "forceConsistentCasingInFileNames": true
  },
  "exclude": [
    "test/"
  ]
}

My test/tests.ts :

import './unit/authentication.js'

Typescript is building my files right. The npm run test cmd returns throw the error I wrote before.

Do you need more context ?

cspotcode commented 4 years ago

@NeilujD this is perfect, thanks.

It looks like, due to missing features in node's ESM support, mocha is using a hack to figure out whether a file should be loaded as ESM or CJS. https://github.com/mochajs/mocha/blob/master/lib/esm-utils.js#L4-L23

ts-node's require() hook will need to be updated to match the error behavior of node's .js hook. When you try to require() a TS file that should be treated as ESM, we should throw an error.

At first I thought mocha could simply import() everything, since it automatically switches to CommonJS loading as needed. However, that would require our ESM hook to be installed in order to resolve and classify .ts files. They're forced to use require() to cater to legacy require() hooks.

cspotcode commented 4 years ago

I think I can hack this by delegating to node's built-in .js extension, passing a fake filename.

require.extensions['.js']({_compile(){}}, filename + 'DOESNOTEXIST.js')

At the cost of a failed fs call, this will cause the built-in require hook to do its package.json lookup and see if the file should be treated as CommonJS or ESM.

> require.extensions['.js'].toString()
'function(module, filename) {\n' +
  "  if (filename.endsWith('.js')) {\n" +
  '    const pkg = readPackageScope(filename);\n' +
  "    // Function require shouldn't be used in ES modules.\n" +
  "    if (pkg && pkg.data && pkg.data.type === 'module') {\n" +
  '      const parentPath = module.parent && module.parent.filename;\n' +
  "      const packageJsonPath = path.resolve(pkg.path, 'package.json');\n" +
  '      throw new ERR_REQUIRE_ESM(filename, parentPath, packageJsonPath);\n' +
  '    }\n' +
  '  }\n' +
  "  const content = fs.readFileSync(filename, 'utf8');\n" +
  '  module._compile(content, filename);\n' +
  '}'

EDIT I've shared this hack with the node folks https://github.com/nodejs/modules/issues/351#issuecomment-625992425 to see if they have any interest in exposing this API natively.

castarco commented 4 years ago

Responding to https://github.com/TypeStrong/ts-node/issues/1007#issue-598417180

Cannot be invoked as ts-node because it requires node flags; hooks cannot be enabled at runtime. This is unavoidable.

Actually... it should be possible, by providing a very simple posix shell script wrapping the whole thing. That would be ideal when using shebangs, as it's not allowed to pass options to shebangs in Linux (although it's possible in Mac).

cspotcode commented 4 years ago

@castarco the problem is cross-platform support that matches the npm ecosystem without adding complexity to ts-node. Typically this is handled by the package manager: npm, yarn, pnpm, etc.

We set up our package.json "bin" field, pointing to a file with a shebang, and it takes care of creating a symlink, .cmd shim, and .ps1 shim.

cspotcode commented 4 years ago

@NeilujD The mocha issue you were seeing should be fixed by #1031 which has been merged to master.

If you're feeling adventurous, you can install ts-node directly from master.

npm i TypeStrong/ts-node#master --save

Or you can download and install the tarball artifact produced by CI.

npm install ts-node-packed.tgz --save

zenflow commented 4 years ago

FWIW I tried it out and got ~~two issues~~ one undocumented issue:

Instructions say to include file extensions in import statements, so that's what I did. But TS complains about this (TS2691: An import path cannot end with a '.ts' extension. Consider importing '../../src/index' instead). I used a @ts-ignore here, and then was able to import that module, sortof.
~~Wherever npm packages are imported I get the error TS7016: Could not find a declaration file for module 'tcp-port-used'. 'C:/Users/Matt/Documents/dev/http-server-group/node_modules/tcp-port-used/index.js' implicitly has an 'any' type. Try 'npm install @types/tcp-port-used' if it exists or add a new declaration (.d.ts) file containing 'declare module 'tcp-port-used';' even though I have those packages sitting in my node_modules, and I don't get this error when I build with tsc.~~

Anyways, just thought I would give some feedback since it was solicited. Keep up the great work @cspotcode!!

cspotcode commented 4 years ago

@zenflow thanks, much appreciated.

Including file extensions is tricky; you need to include them in the way that typescript wants, which is to include the .js extension, not the .ts extension. This comment explains precisely why TypeScript does things this way. It has to do with maintaining compatibility with the pre-compiled code scenario. https://github.com/nodejs/modules/issues/351#issuecomment-621257543

thatsmydoing commented 4 years ago

I've also been trying to get mocha to work with this and I've been following the rabbit hole of https://github.com/mochajs/mocha/issues/4267

master works for me, but to have a usable setup I also needed esModuleInterop enabled and I had to run tests with

node --experimental-modules --loader ts-node/esm.mjs node_modules/mocha/lib/cli/cli.js src/**/*.spec.ts

Calling into the mocha internals is a bit ugly, and probably breaks some features but doesn't require patching node_modules at least.

cspotcode commented 4 years ago

@thatsmydoing Thanks for sharing your experience; I'm sure it will help other people, too.

You should be able to omit --experimental-modules because it is implied by --loader.

We merged #1028 a few days ago, so you should be able to omit the .mjs extension if you want.

This simplifies things a bit:

node --loader ts-node/esm node_modules/mocha/lib/cli/cli 'src/**/*.spec.ts'

Unfortunately, node has a lot of work to do before this feels clean. There's no way for us to load our hooks at runtime, which would allow us to perform the equivalent of --loader on your behalf. There's also no good system for composing multiple loader hooks together. Right now, ts-node's hook is doing extra work that ideally should be handled by a robust hook composer.

akbr commented 4 years ago

Thanks for your time on this! Works great, except I seem to be losing the source map support.

For a two-line test.ts:

type Foo = string;
throw new Error("Oh no!");

(without "type": "module" in package.json)

ts-node test.ts
...
Error: Oh no!(test.ts:2:7) 👍

(with "type": "module" in package.json)

node --loader ts-node/esm.mjs ./test.ts
...
Error: Oh no!(test.ts:1:7) 😢

Is there any easy fix?

cspotcode commented 4 years ago

@akbr Good catch, thanks. It looks like files loaded as ESM have file:// URLs instead of paths.

We install source-map-support here: https://github.com/TypeStrong/ts-node/blob/master/src/index.ts#L445-L451

It's a third-party library that handles sourcemaps automatically, rewriting stack traces. We give it access to a cache of TypeScript's compiler output so it can get the sourcemaps. But we're not handling file:// URLs correctly, so source-map-support is not able to get access to the sourcemaps.

Created #1060 to track this. If you feel like sending us a bugfix, that'd be awesome!

aelbore commented 4 years ago

thanks for the this initiative. I experiment esm and cjs feature heres my library package.json

  "type": "module",
  "main": "./cjs/my-lib.js",
  "module": "./my-lib.js",
  "typings": "./my-lib.d.ts",
  "exports": {
    "require": "./cjs/my-lib.js",
    "import": "./my-lib.js"
  },

if i combine esm and cjs in my code to call the "my-lib" it always use the cjs

import { addNumber } from 'my-lib'
const lib = require('my-lib')

console.log(lib.addNumber(1,2) + addNumber(1,2))

my tsconfig.json

{
  "compilerOptions": {
    "baseUrl": ".",
    "moduleResolution": "node",
    "module": "commonjs",
    "target": "es2018",
    "allowJs": true
  }
}

by this configuration and example it is possible to always call or use the esm module or when there require it uses the cjs then when there import it use the esm?

ExE-Boss commented 4 years ago

@aelbore Since your tsconfig.json file has module: "CommonJS", it will always require(…) the CJS files.

To import the ESM files, you need to set module: "ES2015", module: "ES2020" or module: "ESNext".

Also, you need to construct the require function in ESM modules yourself if you intend to use it:

import { createRequire } from 'module';
const require = createRequire(import.meta.url);

nettybun commented 4 years ago

Is there an easy answer to why __dirname is undefined? Easy to work around but feels unusual. Maybe it could be polyfilled as

import path from 'path';
const __dirname = path.dirname(new URL(import.meta.url).pathname);

As I saw here: https://techsparx.com/nodejs/esnext/dirname-es-modules.html

node --loader ts-node/esm.mjs ./ssr.ts
(node:117914) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
ReferenceError: __dirname is not defined
    at new FileResourceLoader (file:///home/today/_/work/ssr.ts:11:43)
    at file:///home/today/_/work/ssr.ts:26:24
    at ModuleJob.run (internal/modules/esm/module_job.js:110:37)
    at Loader.import (internal/modules/esm/loader.js:179:24)

cspotcode commented 4 years ago

Check node's docs which explain the differences between the ESM and CJS contexts.

On Thu, Jun 11, 2020, 7:47 PM Gen Hames notifications@github.com wrote:

Is there an easy answer to why __dirname is undefined? Easy to work around but feels unusual. Maybe it could be polyfilled as

import path from 'path'; const __dirname = path.dirname(new URL(import.meta.url).pathname);

As I saw here: https://techsparx.com/nodejs/esnext/dirname-es-modules.html

node --loader ts-node/esm.mjs ./ssr.ts (node:117914) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time (Use node --trace-warnings ... to show where the warning was created) ReferenceError: _dirname is not defined at new FileResourceLoader (file:///home/today//work/ssr.ts:11:43) at file:///home/today/_/work/ssr.ts:26:24 at ModuleJob.run (internal/modules/esm/module_job.js:110:37) at Loader.import (internal/modules/esm/loader.js:179:24)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/TypeStrong/ts-node/issues/1007#issuecomment-642985145, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC35OEQUO7UIWHDS4ET43TRWFUHNANCNFSM4MGJCWPA .

ejose19 commented 4 years ago

@cspotcode is there a way to use transpile-only mode? I'm trying to run this in a ts-node-dev way and closest I've got is running nodemon along with -x "node --loader ...", but it's still throwing in type errors instead of just ignoring them (like tsnd does)

cspotcode commented 4 years ago

Set it in your tsconfig file. The SchemaStore schema for tsconfig, which is used for tabcompletion in modern editors, includes the ts-node options.

Environment variables should work, too

On Sat, Jun 20, 2020, 7:22 PM ejose19 notifications@github.com wrote:

@cspotcode https://github.com/cspotcode is there a way to use transpile-only mode? I'm trying to run this in a ts-node-dev way and closest I've got is running nodemon along with -x "node --loader ...", but it's still throwing in type errors instead of just ignoring them (like tsnd does)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/TypeStrong/ts-node/issues/1007#issuecomment-647056399, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC35OGBGLLUZ7KIJA454CDRXVAFBANCNFSM4MGJCWPA .

ejose19 commented 4 years ago

@cspotcode It worked, thanks! trying this on a non-prod project and so far is has been good, excluding the annoyances of using destructured imports, but besides from that all is working (even TLA), will try to integrate jest as well.

Scripts if anyone is interested for faster testing:

"scripts": {
    "start": "node --loader ts-node/esm --experimental-specifier-resolution=node --experimental-top-level-await --no-warnings src/server.ts",
    "dev": "nodemon -q -e ts,js,json -x npm start"
  }

koshic commented 4 years ago

@cspotcode , node-esm-resolve-implementation doesn't support '--experimental-specifier-resolution=node' flag specified trough NODE_OPTIONS env variable because it looks into process.execArgv. Issue can be fixed via '-r module-which-modifies-process-exec-argv.js", but it's really ugly hack.

cspotcode commented 4 years ago

@VladimirGrenaderov Thanks, can you file this as an issue to help track implementation of a fix?

koshic commented 4 years ago

@cspotcode looks a bit confusing:

code from my node_modules (latest version, 8.10.2) - line commented:

https://github.com/TypeStrong/ts-node/blame/master/raw/node-esm-resolve-implementation.js

Somebody change the code before publish?

Because https://registry.npmjs.org/ts-node/-/ts-node-8.10.2.tgz contains commented line too.

cspotcode commented 4 years ago

Make sure you're looking at the correct version tag in GitHub, matching the npm version you're looking at.

On Sat, Jul 4, 2020, 2:36 PM Vladimir Grenaderov notifications@github.com wrote:

@cspotcode https://github.com/cspotcode looks a bit confusing:

code from my node_modules (latest version, 8.10.2) - line commented: [image: image] https://user-images.githubusercontent.com/20106607/86518838-42219100-be3d-11ea-8406-1031ed5e382c.png

https://github.com/TypeStrong/ts-node/blame/master/raw/node-esm-resolve-implementation.js [image: image] https://user-images.githubusercontent.com/20106607/86518851-6a10f480-be3d-11ea-8f0f-bfe7ed97b474.png [image: image] https://user-images.githubusercontent.com/20106607/86518861-79903d80-be3d-11ea-9bdd-7a3278c76a31.png

Somebody change the code before publish?

Because https://registry.npmjs.org/ts-node/-/ts-node-8.10.2.tgz contains commented line too.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/TypeStrong/ts-node/issues/1007#issuecomment-653798204, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC35OEEFUYBF6TT2ZU3LU3RZ5ZDJANCNFSM4MGJCWPA .

koshic commented 4 years ago

yarn info

npm view

Looks like tarball url is correct.

Tag v8.10.2 - https://github.com/TypeStrong/ts-node/blob/v8.10.2/raw/node-esm-resolve-implementation.js, line is not commented:

cspotcode commented 4 years ago

Make sure you're looking at the correct git tag.

On Sat, Jul 4, 2020, 2:49 PM Vladimir Grenaderov notifications@github.com wrote:

yarn info [image: image] https://user-images.githubusercontent.com/20106607/86519048-aba29f00-be3f-11ea-8c2a-de905106aef6.png [image: image] https://user-images.githubusercontent.com/20106607/86519068-ddb40100-be3f-11ea-9ae0-1a8f2cf48b82.png

npm view [image: image] https://user-images.githubusercontent.com/20106607/86519081-06d49180-be40-11ea-8690-9eedc61fdccb.png

Looks like tarball url is correct.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/TypeStrong/ts-node/issues/1007#issuecomment-653799222, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC35OBUC362GGC4MVWPRHDRZ52RZANCNFSM4MGJCWPA .

koshic commented 4 years ago

@cspotcode yep I catch it - dist-raw/node-esm-resolve-implementation.js & raw/node-esm-resolve-implementation.js are different. Thx, but what does it mean? Just experimental garbage?

As you asked, new issue - https://github.com/TypeStrong/ts-node/issues/1072.

sionzee commented 4 years ago

https://github.com/TypeStrong/ts-node/issues/1007#issuecomment-631205443

The node --loader ts-node/esm node_modules/mocha/lib/cli/cli 'src/**/*.spec.ts' is not working in this case because the esm.mjs file has require for

const esm = require('./dist/esm')

but the folder dist doesn't exist when it is "installed" from TypeStrong/ts-node#master

A small hack is going to node_modules/ts-node and type npm install + npm run build-nopack

But if user doesn't specifiy --experimental-specifier-resolution=node the ts-node will start complaining about ERR_MODULE_NOT_FOUND. Because the normal resolution is not resolving missing .ts extension.

Eg.:

import {Something} from "../src/shared/TestFile";

Complains:

Error: ERR_MODULE_NOT_FOUND /src/shared/TestFile /tests/TestFile.spec.ts module
at finalizeResolution (/node_modules/ts-node/dist-raw/node-esm-resolve-implementation.js:360:9)

So the solution is having the specifier.

"test": "node --loader ts-node/esm --experimental-specifier-resolution=node node_modules/mocha/lib/cli/cli --config _config/.mocharc.cjs"

When do you plan next release of ts-node?

cspotcode commented 4 years ago

The dist folder should be created when you install from master using npm, because npm should be running the build script. CI produces a packaged tarball artifact which should also include the dist folder. Downloading the archive that GitHub generates -- the one that only contains the contents of master -- will likely not work.

On Sun, Jul 5, 2020, 9:32 AM sionzeecz notifications@github.com wrote:

@NeilujD https://github.com/NeilujD The mocha issue you were seeing should be fixed by #1031 https://github.com/TypeStrong/ts-node/pull/1031 which has been merged to master.

If you're feeling adventurous, you can install ts-node directly from master.

npm i TypeStrong/ts-node#master --save

Or you can download and install the tarball artifact produced by CI. [image: image] https://user-images.githubusercontent.com/376504/82399938-32ccbc80-9a24-11ea-93b7-52c457617a51.png [image: image] https://user-images.githubusercontent.com/376504/82399965-3c562480-9a24-11ea-9ec6-3d65616e69e6.png

npm install ts-node-packed.tgz --save

The node --loader ts-node/esm node_modules/mocha/lib/cli/cli 'src/*/.spec.ts' is not working in this case because the esm.mjs file has require for

const esm = require('./dist/esm')

but the folder dist doesn't exist when it is "installed" from TypeStrong/ts-node#master

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/TypeStrong/ts-node/issues/1007#issuecomment-653889543, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC35OCMGBMXGRMHABJDBL3R2B6GTANCNFSM4MGJCWPA .

sionzee commented 4 years ago

I have used yarn.

yarn add --dev TypeStrong/ts-node#master

seems it hasn't called the prepare event. https://github.com/yarnpkg/yarn/issues/1671

Sorry for false fire!

yarn add --dev git+https://github.com/TypeStrong/ts-node Works as is expected.

cspotcode commented 4 years ago

@sionzeecz Thanks for sharing; I'm sure this will help others using yarn, too.

KilianKilmister commented 4 years ago

for the time being, could we include a note about adding additional type-roots?

since ts-node --files isn't an options here, people (including me up until 5min ago) might not know how to do it otherwhise.

something like

note: to include typeroots specified in tsconfig, add env var TS_NODE_FILES=true

as a hashbang: #!/usr/bin/env TS_NODE_FILES=true node --loader ts-node/esm.mjs note hasbang arguments arent available on some unix distros (including linux). it works on macOS for example.

besides that i't's generally been working very well and made coding ts+esm a much smoother experience

KilianKilmister commented 4 years ago

EDIT: whoops, i missed the note for it:

Set "type": "module" in your package.json, which is required to tell node that .js files are ESM instead of CommonJS. To be compatible with editors, the compiler, and the TypeScript ecosystem, we cannot name our source files .mts nor .mjs.

have you considered adding support for .mts/.ctsextentions? i don't know how that would be handled by standard ts-compiler and wether that could be added as a compiler-plugin if not.

this isn't really nescessary of course, i just wanted to add a note about it

cspotcode commented 4 years ago

I have. To elaborate on the explanation that you found:

The problem is it makes code incompatible with the TypeScript ecosystem. Try importing from a .mts or .cts file and see what the language service says.

The benefits of TypeScript largely come from the language service. Even if you may individually disagree with that statement, we can't be messing things up for people who do benefit greatly from the language service.

Additionally, runtime compiling in production doesn't make sense for obvious reasons. Again, some people may individually agree with this statement, which is fine, but we can't be encouraging code that's deliberately incompatible with precompilation.

EDIT: it'll be interesting to see if Deno causes any changes in the ecosystem, since they use the .ts file extension in their import specifiers, and they've had to write editor integrations to somehow modify behavior of the language service.

KilianKilmister commented 4 years ago

The problem is it makes code incompatible with the TypeScript ecosystem. Try importing from a .mts or .cts file and see what the language service says.

never tried it, but i thougth so. that why i added the note about a compiler-plugin. there is a plugin system, but it's not used much and i have no idea how it works. was just a thought

The benefits of TypeScript largely come from the language service. Even if you may individually disagree with that statement, we can't be messing things up for people who do benefit greatly from the language service.

100% agree. my code style is basically vanilla-esm with type anotations and my default compile target is ESNEXT, so the language feature is almost the only reason why i use ts

The reason why i wondered is because of this vanilla-sentiment. it would allow for the emitted files to be even closer to the source. It's just something i like, to have that smooth transition and it makes compiler-side oddities and mistakes a breeze to track down. it's just satisfying to me how together with the typedeclarations that have the same vanilla-look, it merges together into prettymuch exactly what the source-code is on a visual level aswell.

EDIT: it'll be interesting to see if Deno causes any changes in the ecosystem, since they use the .ts file extension in their import specifiers, and they've had to write editor integrations to somehow modify behavior of the language service.

I haven't looked at deno in quite a while. i followed it's development early-on but i'm quite happy with node, so for now i don't see a personal need to use it. but it currently looks like it's here to stay, and some competition can cause benefitial changes on both sides. I'm really looking forward to what developments will happen in nodes vm.module and policies. i think deno will have some influance on them.

cspotcode commented 4 years ago

The TypeScript compiler intentionally enforces that extensions in import statements include the .js extension, because this makes the import statement in your source code exactly match the emitted import statement. import {foo} from './foo.js'; will be emitted. At compile-time it's looking at foo.ts. At runtime it's looking at foo.js. The import statement is identical both places.

MicahZoltu commented 4 years ago

This doesn't appear to work with mixed module type dependencies. I have a dependency that has the following in its package.json:

"main": "./output-cjs/index.js",
"exports": {
    "import": "./output-esm/index.js",
    "require": "./output-cjs/index.js"
},

When I am in a project that references it and do

node --loader ts-node/esm.mjs --experimental-specifier-resolution=node tests/index.ts

I get an error:

The requested module '@zoltu/rlp-encoder' is expected to be of type CommonJS, which does not support named exports. CommonJS modules can be imported by importing the default export.

Why is it expecting the dependency package to be CJS? I am specifically trying to load everything as ESM modules. Both CJS and ESM are available, so why is failing to load whichever one it is trying to load?

cspotcode commented 4 years ago

Without a full reproduction, I'm guessing it might be an accident in the way the package.json is written. It looks slightly different than the examples from node's documentation. https://nodejs.org/api/esm.html

We can dig in further if we get a full reproduction that proves the problem occurs with ts-node and not with TSC.

On Sat, Jul 25, 2020, 8:45 AM Micah Zoltu notifications@github.com wrote:

This doesn't appear to work with mixed module type dependencies. I have a dependency that has the following in its package.json:

"main": "./output-cjs/index.js","exports": { "import": "./output-esm/index.js", "require": "./output-cjs/index.js" },

When I am in a project that references it and do

node --loader ts-node/esm.mjs --experimental-specifier-resolution=node tests/index.ts

I get an error:

The requested module '@zoltu/rlp-encoder' is expected to be of type CommonJS, which does not support named exports. CommonJS modules can be imported by importing the default export.

Why is it expecting the dependency package to be CJS? I am specifically trying to load everything as ESM modules. Both CJS and ESM are available, so why is failing to load whichever one it is trying to load?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/TypeStrong/ts-node/issues/1007#issuecomment-663851595, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC35OEJPMTPBJFID2M7TB3R5LHXFANCNFSM4MGJCWPA .

MicahZoltu commented 4 years ago

Repro: https://github.com/MicahZoltu/eip-2718-tests/tree/ts-node-repro

This is the first project I've tried to use with ts-node & ESM, so I may be doing something wrong.

cspotcode commented 4 years ago

@MicahZoltu thanks, having a complete example makes debugging straightforward.

I tried the following with node v14.2.0 on Linux:

# At the shell prompt
$ npm install @zoltu/rlp-encoder # EDIT: installed v2.0.3
$ node
# in the node REPL
# Ask node what it wants to load for CommonJS.  This part looks good.
> require.resolve('@zoltu-rlp-encoder')
..../repros/node_modules/@zoltu/rlp-encoder/output-cjs/index.js'
# Ask node to load it as ESM
> import('@zoltu/rlp-encoder')
Promise { <pending> }
> (node:5815) UnhandledPromiseRejectionWarning:..../repros/node_modules/@zoltu/rlp-encoder/output-esm/index.js:1
export function rlpEncode(item) {
^^^^^^

SyntaxError: Unexpected token 'export'
    at Object.compileFunction (vm.js:344:18)

Based on the above, and without using ts-node at all, it looks like node is trying to load the ESM files as CommonJS.

This makes sense because the file extension is .js, so node will use package.json to figure out how to load the file. This tells node to load it as CommonJS.

So it looks like the problem is unrelated to ts-node and is caused by a faulty build process for @zoltu/rlp-encoder.

MicahZoltu commented 4 years ago

Hmm, thanks for looking into it, I'll dig a bit more. Looking at the script output you provided, it does appear to be loading the correct index file (it is using the exports property in package.json to figure out paths), but it then proceeds to complain that export is not a valid token in an ESM.... (which it definitely is) which is very bizarre.

cspotcode commented 4 years ago

The node docs call out this behavior explicitly, so it's not a node bug. https://nodejs.org/api/esm.html#esm_conditional_exports

MicahZoltu commented 4 years ago

The confusion from the docs was here: https://nodejs.org/api/esm.html#esm_dual_commonjs_es_module_packages

Node.js can now run ES module entry points, and a package can contain both CommonJS and ES module entry points (either via separate specifiers such as 'pkg' and 'pkg/es-module', or both at the same specifier via Conditional exports).

However, if I set type: 'module' it only works with import ... and if I don't set type: 'module' then it only works with require(...). I'll take this up with the NodeJS people though, since it sounds like the issue isn't with ts-node.

MicahZoltu commented 4 years ago

For any future readers wanting to follow along on my journey toward functional dual module packages: https://github.com/nodejs/node/issues/34515

TypeStrong / ts-node

ESM support: soliciting feedback #1007

Experimental warning

Third-party docs: "Guide: ES Modules in NodeJS"

First-party docs

Usage

Requirements

Invocation

Configuration

Versioning

node's API change: v16.12.0, v17.0.0

Pending development work

The proposal

Usage

ESM hooks

`resolve()` hook:

`getFormat` hook:

`transformSource` hook:

Changes to existing functionality

`require()` hook

`require()` code transform

`ts-node` bin entry-point

Allow `import()` in CJS

Emit considerations

Set "type": "module" in your package.json, which is required to tell node that .js files are ESM instead of CommonJS. To be compatible with editors, the compiler, and the TypeScript ecosystem, we cannot name our source files .mts nor .mjs.

TypeStrong / ts-node

ESM support: soliciting feedback #1007

Experimental warning

Third-party docs: "Guide: ES Modules in NodeJS"

First-party docs

Usage

Requirements

Invocation

Configuration

Versioning

node's API change: v16.12.0, v17.0.0

Pending development work

The proposal

Usage

ESM hooks

resolve() hook:

getFormat hook:

transformSource hook:

Changes to existing functionality

require() hook

require() code transform

ts-node bin entry-point

Allow import() in CJS

Emit considerations

Set "type": "module" in your package.json, which is required to tell node that .js files are ESM instead of CommonJS. To be compatible with editors, the compiler, and the TypeScript ecosystem, we cannot name our source files .mts nor .mjs.

`resolve()` hook:

`getFormat` hook:

`transformSource` hook:

`require()` hook

`require()` code transform

`ts-node` bin entry-point

Allow `import()` in CJS