New `--import` API breaks worker threads

zner0L commented 1 year ago

Problem

I was tracking https://github.com/esbuild-kit/tsx/issues/330 closely and was very happy for you to fix it with https://github.com/esbuild-kit/tsx/pull/337 and release so quickly, but I just now tested it with loading worker threads and that fails with ERR_UNKNOWN_FILE_EXTENSION.

Minimal example:

import { isMainThread, Worker } from 'worker_threads';

if (isMainThread) {
    const worker = new Worker(new URL(import.meta.url));
} else console.log('Hello from worker thread!');

Result:

yarn tsx test.ts
yarn run v1.22.19
$ /<redacted>/node_modules/.bin/tsx test.ts

node:internal/event_target:1083
  process.nextTick(() => { throw err; });
                           ^
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /<redacted>/test.ts
    at __node_internal_captureLargerStackTrace (node:internal/errors:497:5)
    at new NodeError (node:internal/errors:406:5)
    at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:99:9)
    at defaultGetFormat (node:internal/modules/esm/get_format:142:36)
    at defaultLoad (node:internal/modules/esm/load:120:20)
    at ModuleLoader.load (node:internal/modules/esm/loader:396:13)
    at ModuleLoader.moduleProvider (node:internal/modules/esm/loader:278:56)
    at new ModuleJob (node:internal/modules/esm/module_job:65:26)
    at #createModuleJob (node:internal/modules/esm/loader:290:17)
    at ModuleLoader.getJobFromResolveResult (node:internal/modules/esm/loader:248:34)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:229:17)
    at async ModuleLoader.import (node:internal/modules/esm/loader:315:23)
    at async loadESM (node:internal/process/esm_loader:34:7)
    at async handleMainPromise (node:internal/modules/run_main:66:12)
Emitted 'error' event on Worker instance at:
    at [kOnErrorMessage] (node:internal/worker:326:10)
    at [kOnMessage] (node:internal/worker:337:37)
    at MessagePort.<anonymous> (node:internal/worker:232:57)
    at [nodejs.internal.kHybridDispatch] (node:internal/event_target:807:20)
    at exports.emitMessage (node:internal/per_context/messageport:23:28) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}

Node.js v20.8.0

Expected behavior

If I run the minimal example with tsx v3.13.0 the output is as expected (though with warnings):

❯ yarn tsx test.ts
yarn run v1.22.19
$ /<redacted>/node_modules/.bin/tsx test.ts
(node:35991) ExperimentalWarning: `--experimental-loader` may be removed in the future; instead use `register()`:
--import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("file%3A///<redacted>/node_modules/tsx/dist/loader.mjs", pathToFileURL("./"));'
(Use `node --trace-warnings ...` to show where the warning was created)
(node:35991) ExperimentalWarning: `--experimental-loader` may be removed in the future; instead use `register()`:
--import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("file%3A///<redacted>/node_modules/tsx/dist/loader.mjs", pathToFileURL("./"));'
(Use `node --trace-warnings ...` to show where the warning was created)
Hello from worker thread!
Done in 0.25s.

Minimal reproduction URL

Included in issue

Version

v3.14.0

Node.js version

v20.8.0

Package manager

yarn

Operating system

Linux

Contributions

[ ] I plan to open a pull request for this issue
[ ] I plan to make a financial contribution to this project

ghost commented 1 year ago

I tried passing options to the Worker constructor such as env: { NODE_OPTIONS: "--import tsx" } and execArgv: ["--import", "tsx"] but neither of those worked and then I found nodejs/node#47747 (ESM loaders cannot be defined via Worker option...)

I think this is really a Node bug or design issue. If a loader was specified at the top level, Node should be using that automatically for Worker threads.

ghost commented 1 year ago

Other notes

Node has a plan to support identifying external modules, e.g. typescript see nodejs/node#49704
There is a discussion about missing use cases for the module.register() API at nodejs/node#50392
I found the above 2 notes from a comment in a proposal to use the ESM loader everywhere, see https://github.com/nodejs/node/issues/50356#issuecomment-1779592133

vafanassieff commented 1 year ago

I'm facing the same issue as you do, and I had to stick with tsx v3.13.0

alanhoff commented 1 year ago

I've found a temporary workaround: while developing, you'll want your worker to load tsx/cli instead, then pass the actual script you wanna run as an argv, like so:

    const tsx = new URL(import.meta.resolve('tsx/cli'));
    const worker = new Worker(tsx, {env: SHARE_ENV, argv: ['./your/script.ts']});

privatenumber commented 1 year ago

Looked into this. Sounds like this is a limitation on Node's end? Marking this as blocked

SynthLuvr commented 11 months ago

This is an issue for node v18.19.0 too

nekochan0122 commented 11 months ago

I got an error with this Error: listen EADDRINUSE: address already in use.

I've found a temporary workaround: while developing, you'll want your worker to load tsx/cli instead, then pass the actual script you wanna run as an argv, like so:
  const tsx = new URL(import.meta.resolve('tsx/cli'));
  const worker = new Worker(tsx, {env: SHARE_ENV, argv: ['./your/script.ts']});

privatenumber commented 11 months ago

🛑 Want this fixed?

This is currently a limitation in Node.js. The issue was closed without resolution: https://github.com/nodejs/node/issues/47747

They currently don't think the performance cost is worth it, but are open to learning about more use-cases.

I have requested for it to be re-opened. If you'd like loaders to be supported in Workers,

your best course of action is to share your use-case there.

Again, don't just say "plz fix". Share your use-case in https://github.com/nodejs/node/issues/47747.

vafanassieff commented 11 months ago

Our use case is running tsx --watch as dev server for a typescript project who use a Worker for specific tasks and run tests with vitest

For now the only fix we found was to stick with node 20.9.0 and tsx 3.13.0 We have to start our tests script with NODE_OPTIONS='--loader tsx --no-warnings' vitest --run

SynthLuvr commented 11 months ago

Our use case is using tsx with ava. Similarly, we have to downgrade for now.

JounQin commented 11 months ago

I'm not using any worker threads, but just:

import fs from 'node:fs'

import LinguistLanguages from 'linguist-languages'
import { SupportLanguage } from 'prettier'

https://github.com/un-ts/prettier/actions/runs/7152253084/job/19477387552?pr=327#step:5:96

See source codes at https://github.com/un-ts/prettier/blob/master/scripts/languages.ts

privatenumber commented 11 months ago

This is blocked by Node.js. Please contribute constructively in https://github.com/nodejs/node/issues/47747

Looks like it's been re-opened with a potential solution that could use help.

Locking thread until resolved in Node.

SynthLuvr commented 3 weeks ago

https://github.com/nodejs/node/issues/47747 was closed without fix

privatenumber commented 3 weeks ago

I'm going to lock this issue again to avoid unproductive comments and encourage actionable steps.

Since the root problem lies within Node, the conversation will have to happen there. If the Node issue was closed without a resolution, I suggest filing a new one with a clear, reproducible example to demonstrate how the problem still exists.

privatenumber / tsx