nodejs / node

Node.js JavaScript runtime ✨🐢🚀✨
https://nodejs.org
Other
108.05k stars 29.83k forks source link

How could we support typescript without vendoring it? #43818

Open mcollina opened 2 years ago

mcollina commented 2 years ago

I would like Node.js to support the following user experience

$ node script.ts
Typescript support is missing, install it with:
npm i --location=global typescript-node-core
$ npm i --location=global typescript-node-core
...
$ node script.ts
"Hello World"

(I picked the typescript-node-core name at random, I would be extremely happy if that could be ts-node)

Note that script.ts could evaluate to either cjs or mjs depending on the tsconfig.json file.

Originally posted by @mcollina in https://github.com/nodejs/node/issues/43408#issuecomment-1182883689


Why vendoring is not an option?

  1. TypeScript changes too fast and they do not follow semantic versioning, nor do they provide any long-term support.
  2. We will never be able to pick a default tsconfig given the 100+ options.
JakobJingleheimer commented 2 years ago

I'm interested and can attend that time on either day.

aduh95 commented 2 years ago

10-12 ET

~Morning on the US east coast is night time in Europe; if there's a time slot compatible with Europe time zone~ EDIT: I mixed up the timezones, nevermind. I'd be interested :)

mhdawson commented 2 years ago

@aduh95 10-12 ET is 3-5 pm in France I think. It's alreay starting at 7 AM for anybody in pacific time (west coast NA) so seem like a reasonable balance between West coast North America and Europe ?

mcollina commented 2 years ago

That would work for me

kdy1 commented 2 years ago

Am I invited too?

Not sure because of my English skill

Qard commented 2 years ago

I'll try to attend, if I can. 🙂

giltayar commented 2 years ago

Interested. Both times would work for me.

meyfa commented 2 years ago

Would it be possible to attend this meeting as a silent spectator? I'm not affiliated with Node.js or TypeScript, but deeply curious how the process works and what gets decided on the topic.

mhdawson commented 2 years ago

@meyfa the meeting will be open and you are welcome to come and listen/participate.

mhdawson commented 2 years ago

@kdy1 invitation is open.

Ethan-Arrowood commented 2 years ago

Can attend - multiple other Vercel folks may attend as well. Once a meeting invite is created I'll be sure to share internally

mhdawson commented 2 years ago

Created issue for a mini-summit here https://github.com/nodejs/next-10/issues/149

alangpierce commented 2 years ago

Hi, I'm the author of Sucrase. I won't be able to make the meeting, but am definitely interested in DX improvements here and wanted to leave a few thoughts:

cspotcode commented 2 years ago

@alangpierce that's great to hear about the Sucrase plugin. Can we sync up about your plans, either by talking in a sucrase issue or a ts-node issue? I had been worried about biting off more than I can chew with implementing additional transpiler plugins myself, but Sucrase was on my list because it fits a need for a smaller pure-JS option. So if there's anything you need from my end to make it a success, I want to help.

jakebailey commented 2 years ago

I mentioned in the call some projects which have the ability to use TS as config files, but where that experience is difficult, and @GeoffreyBooth asked me to post some of that info here.

Webpack does (and it's a lot of instructions). That uses interpret and rechoir, which let you require files of other types.

vite supports vite.config.ts by directly calling esbuild.

I couldn't remember the name of the library I was thinking of that lets people use .config.ts files more generically (it wasn't rechoir), but you can see more examples by doing a code search for config files with TS extensions.

EDIT: I was thinking of rc-config-loader and cosmiconfig.

cspotcode commented 2 years ago

This is a good example of where node can offer better first-class support. Interpret, rechoir, etc use CommonJS require.extensions hooks, so they don't work for ESM. More generally, they assume the ability to install hooks in-process. This has fragmented how tooling (the consumers of rechoir) have decided to support ESM.

For example, I believe some that call out to esbuild are actually bundling the config file with its deps, writing it to a temp location on disk, and then executing that script in-process.

Others decide to spawn a child process with --loader for the config file, then marshal the config object across process boundaries as JSON. Webpack config can't do that since the config file has executable new PluginClass() stuff in it.

This potentially relates to how we support hook configuration in package.json. node ./foo.ts can pull foo's package.json But mocha ./foo.spec.ts is calling an entrypoint that's outside of the project, which is in turn trying to execute code inside the project.

I was distracted by work at the tail-end of the call, so perhaps these concerns were already noted.

I couldn't remember the name of the library I was thinking of that lets people use .config.ts files more generically

Maybe liftoff?

privatenumber commented 2 years ago

Just stumbled across this thread and wanted to share TypeScript Execute (tsx), which I think offers a very similar developer experience described by OP:

$ npx tsx script.ts
"Hello World"

# If globally installed
$ tsx script.ts
"Hello World"

It's zero-config (no tsconfig.json necessary) and automatically accommodates CommonJS/Module package contexts.

It's powered by esbuild and uses @esbuild-kit/esm-loader (Node.js loader) & @esbuild-kit/cjs-loader (require() hook) under the hood.

I would love to see further TypeScript integration in Node.js, and I think tsx demonstrates the demand and practicality for tools like it.

JakobJingleheimer commented 2 years ago

Mm, I stumbled upon tsx recently. Not to digress: I couldn't figure out how it's actually different from ts-node (it seems the only difference is its claim to be faster than ts-node because it uses esbuild (go) whereas ts-node uses tsc which is JavaScript, but that's not really true—ts-node has an swc (rust) compiler option).

Advocates seeking more support with TypeScript have explicitly stated that somePackage script.ts is too much trouble for end-users, so the example of tsx script.ts unfortunately doesn't fit the bill (something like that was an early suggestion).

The no-config part might appeal though (I think esbuild itself does not require a tsconfig?).

privatenumber commented 2 years ago

I elaborated on the comparison in the FAQ. Hope it clarifies the differences—happy to answer questions if you open a discussion in the repo.

I shared tsx for those that aren't convinced that Node.js's DX can improve dramatically by supporting a zero-config TypeScript runtime. Perhaps they can be persuaded by trying it.

MoLow commented 2 years ago

where does this stand? is there anyone working on this / anyplace where this is tracked?

GeoffreyBooth commented 2 years ago

where does this stand?

Per https://github.com/nodejs/next-10/issues/149, the document https://github.com/nodejs/node/blob/main/doc/contributing/maintaining-types-for-nodejs.md was updated with the consensus next steps: https://github.com/nodejs/node/commit/dcc9589e5ed36d511466fd43ea1f4b9c1f567899.

is there anyone working on this / anyplace where this is tracked?

Not that I’m aware of, but you can comment here (or open a new issue) if you’d like to start implementing some of the steps listed in https://github.com/nodejs/node/blob/main/doc/contributing/maintaining-types-for-nodejs.md#high-level-approach—development-workflow and we can coordinate effort.

JakobJingleheimer commented 2 years ago

I can update the format error message; the doc it cites needs to exist first though.

If the file was a TypeScript file, a TypeScript specific message with a reference to a link on Nodejs.org specific on learning how to configure TypeScript will be provided.

I'm think a subsequent enhancement to this could be to "register" file extensions with their specific doc entry if it exists, and use that decide whether to show a generic or specific link

Ex

if (isUnsupportedFormat(url)) {
  const link = thirdPartyFormats.get(ext) || genericLink;

  const message = `…`;
}
Qard commented 2 years ago

We could give it its own error code and add it to the existing error code list: https://nodejs.org/api/errors.html#nodejs-error-codes

That would give us a path to eventually make unique pages per error code with more detailed and up-to-date information.

orta commented 2 years ago

Hey folks, I was in openjs summit's node.js tooling group session yesterday where this came up and tried to come up with a new idea for how this could work.

To set up my priors, there's a few reasonable tensions at play:


I think a potential answer for out of the box index.ts support in node.js is that the "typescript" npm module adds support for the ESM loader APIs, and node.js adds support which looks to see if "typescript" is installed locally via node_modules (or globally) and automatically run the esm loaders for *.ts, *.tsx, *.mts and *.cts.

It puts some onus on both projects to move a little bit, but neither need to be fully committed to a vendored-ish connection where they are blocking each others work.

If the "typescript" lookup fails, you can recommend someone install "typescript" via npm. Given the need/desire to type-check *.ts files, it's pretty likely this is already set up.


The pile of caveats:

Note, that I'm not saying that adding ESM loaders support is trivial for the TS team - but it is reasonably sized looking at the size of the support which was added to ts-node (~400 LOC, which then reaches into a bunch of pre-existing code which was already necessary for prior similar work)

mcollina commented 2 years ago

I'm in favor of @orta proposal. I have some questions:

Does this even work for node index.ts because it requires the files to be ESM meaning there must be a package.json - is that an acceptable trade-off?

Why? I don't see why this would be needed.

Performance oriented JS transpilers (sucrase/swc/esbuild/etc) would likely be faster than the "typescript" transpile APIs, but at the trade-off of perfect accuracy - this doesn't proclude them being used as loaders, but does prefer the built-in TypeScript. It feels like a push to say that node could have a list of their loaders are prefer them, but I think there'd be resistance in node.js to that given the dancing landscape of the JS ecosystem.

We should provide some configurability for this mechanism, something in package.json that allows us to map .ts to the swc loader.

orta commented 2 years ago

I assumed that because they were called ESM module loaders they would be ESM only - but if not, that makes a stronger case for this technique.

The configurability makes sense, as "typescript" would be default any other loader-ish package definition should be able to overwrite it 👍🏻

mcollina commented 2 years ago

I assumed that because they were called ESM module loaders they would be ESM only - but if not, that makes a stronger case for this technique.

I think there is a plan / possibility to extend them to CJS too.

GeoffreyBooth commented 2 years ago

Hey folks, I was in openjs summit’s node.js tooling group session yesterday where this came up and tried to come up with a new idea for how this could work.

Hi @orta, thanks for your ideas. A few thoughts:

Loaders/Hooks/Customizations are still being figured out but having this as a launch goal could be a big ecosystem win, and could help shape their API?

Loaders are on the verge of being stable: https://github.com/nodejs/loaders#milestone-1-parity-with-commonjs. Adding support for some type of file-based configuration (as opposed to the current flag-based) is on the roadmap, but unnecessary for the Loaders API to leave experimental status because it shouldn’t be a breaking change to add a new way of defining configuration. There are no API changes needed to support TypeScript, as it works today via things like node --loader ts-node/esm. All that remains is making this easier, either through prompting the user with what to do or automating some or all of the steps to making it work.

Also, a solution along these lines, where we don’t special-case the typescript package, means that we don’t need any buy-in from the TypeScript team. We could solve this all on our own, and users could choose whichever TypeScript library they prefer.

cspotcode commented 2 years ago

There are no API changes needed to support TypeScript, as it works today via things like node --loader ts-node/esm.

This is a bit misleading since ts-node/esm installs several non-loader hooks into the main thread. Yes, it does work today, but future changes to node or to the loader API might break or hinder our ability to install these non-loader hooks. From a user's perspective, it's a loss of functionality when those hooks go away: TS that was working previously will start failing with runtime errors.

GeoffreyBooth commented 2 years ago

Yes, it does work today, but future changes to node or to the loader API might break or hinder our ability to install these non-loader hooks.

I’m unsure what this means. We’re not intending to change Node to break TypeScript support. On the contrary, making the Loaders API stable is meant to achieve the opposite: an API that can be relied upon, that won’t be removed or changed absent a Node semver-major change.

We have discussed adding similar customization APIs for things like source maps and the REPL and so on; I assume these are the “non-loader hooks” you refer to? The goal there would be to add APIs to allow tools like ts-node to customize those areas of Node in non-hacky ways that also provide more stability, as they would be officially supported Node APIs rather than unofficial monkey-patching or the like. But again, no one benefits from changes to Node causing breaks in expected behavior. The goal would always be to improve compatibility and reliability.

cspotcode commented 2 years ago

I’m unsure what this means.

We have to use undocumented monkey-patches. When node ships breaking changes to undocumented monkey-patchable functions -- perhaps accidentally -- there is sometimes resistance to fixing the breakage, since the API surface is undocumented. Stabilizing loaders is good, but it doesn't stabilize these undocumented hooks. From a maintainer's perspective, this risk is not fun.

One of those hooks is to give us proper module resolution: Module._resolveFilename It's not related to REPL nor sourcemaps.

I agree everyone's on-board to enable great TS support. I hope that until the loaders API fully supports the necessary hooks, the node team will treat these undocumented monkey-patches as an API surface that cannot be broken.

GeoffreyBooth commented 2 years ago

I hope that until the loaders API fully supports the necessary hooks, the node team will treat these undocumented monkey-patches as an API surface that cannot be broken.

The undocumented hooks you’re talking about are additional scope beyond the Loaders API, but something we intend to address with a new Customization API: https://github.com/nodejs/loaders/issues/95. The goal is to support things like ts-node entirely through documented, stable APIs.

knyga commented 2 years ago

These days it is common to use services that do provisioning and management for you, AWS Lambda in particular. So they do not care that much about node.js runtime. But developers do care. Developers are forced to transpile the code before deployment and have transpiling pipeline.

Secondly, I can't believe external transpilation could do better than native execution.

Thirdly, bun and deno already have this feature. Their runtimes want to stand out and get some audience based on unsolved needs. Isn't it a good sign that market expects TypeScript to be there?

eduardofx commented 2 years ago

I think we can have pre-defined values of tsconfig and the developer who wants to use other resources create inside the folder manually.

vdeturckheim commented 1 year ago

Ok, I started to experiment with implementation and here are my findings/opinions for tonight:

I won't have coding time for a couple days, but should go back into writing stuff early next week

GeoffreyBooth commented 1 year ago

@vdeturckheim This looks good! A few thoughts:

vdeturckheim commented 1 year ago

Thanks @GeoffreyBooth , these are very good points! Yeah, @node/tsc-adapter is another name for typescript-node-core suggested in the first message of the thread and would be ts-node with node using it instead of having to plug into node. Basically, making the TS compiler live outside of node while allowing node xx.ts without any other argument. Right now, I made it work with cjs but for esm it will likely be a loader (we had experiments with @targos a few years ago on that too).

Regarding node vs. package version, this is a very good point. I think we are going to a direction where the typescript support can work very well in most cases and where we provide DIY solutions for when it does not (like overriding the typescript adapter for instance).

For repl and .ts cases. I agree 💯 with you. In the end, these are best-effort and DX choices, we can take the decisions and document them. Also, this is what experimental support is for!

I missed last summer's summit (and this thread until recently fwiw), so I hope I am still going in the direction people agreed upon 👼

GeoffreyBooth commented 1 year ago

I missed last summer's summit (and this thread until recently fwiw), so I hope I am still going in the direction people agreed upon 👼

Yes, I think nothing has changed from https://github.com/nodejs/node/issues/43818#issuecomment-1264954915 and what your recent experiments seem to assume. About the only “news” is that we’ve made so much progress with the Loaders API that I can see it applying to CommonJS too in the not-too-distant future; so I would prioritize getting TypeScript support via Loaders working first or at least ship that at the same time as CommonJS solution. @cspotcode and others have also had discussions about hooks in addition to those provided in the Loaders API, to do things like customize the REPL, for a more seamless DX beyond loading modules. Hopefully we can use the interest in better TypeScript support to build out some of those APIs.

It sounds like package.json options aren’t needed for the core/MVP use case, so maybe we should get the initial to-do list done and then we can loop back to that and give it the attention it deserves.

vdeturckheim commented 1 year ago

This sounds good, thanks for the highlights! Seems good to me! I'll resume work with that in mind!

Regarding the point in not treating "typescript" differently (in https://github.com/nodejs/node/issues/43818#issuecomment-1264954915), I agree, that's one of the advantage of having something like @node/tsc-adapter which we control and would only serve the goal of calling "typescript" from node under the hood on .ts files. But nothing prevents to also have @node/swc-adapter or @node/esbuild-adapter tooto let the end user chose the transpiler in a list of provided alternatives (these packages are proxies to the other packages which also abstract breaking changes such packages could have). The end user will anyway be able to use a custom loader if they want to use something else in the end.

But let's focus on the MVP for now and add too much complexity later 👼

tony-go commented 1 year ago

But nothing prevents to also have @node/swc-adapter or @node/esbuild-adapter tooto let the end user chose the transpiler in a list of provided alternatives (these packages are proxies to the other packages which also abstract breaking changes such packages could have). The end user will anyway be able to use a custom loader if they want to use something else in the end.

+1 on this ^^

GeoffreyBooth commented 1 year ago

But nothing prevents to also have @node/swc-adapter or @node/esbuild-adapter tooto let the end user chose the transpiler in a list of provided alternatives

I’m not sure Node needs to publish a transpiler at all. Whatever docs page our error message links to could include a list of ones known to work, and encourage the user to choose one to install. I would think we should at least start with that approach, and if we find a need for publishing something official we could do that after MVP.

hinell commented 1 year ago

TypeScript is very slow, full of silly bugs, weird semantics. Its devs constantly-introduce complex features. Its typed system is a complete joke and doesn't take full advantage of overloading which brings performance in others languages. It also breaks backward compatibility often and simply sucks. Being made by Miscrosoft who are infamous for attempting to lock-in users and dropping their products at will (RIP Atom) is also another major issue over here. Important bugs in TS remain open for years and nobody cares about them. Choosing TS is not best choice.

We need a better typed language without all that complexity fuss.

See my post over here: https://gist.github.com/hinell/feb13b6a583f8c2a6734d8d881a45e06

hinell commented 1 year ago

There is a related proposal to add type syntax to JS, checkout:

shinebayar-g commented 11 months ago

What is the latest update on this? With both Deno and Bun improving their compatibility with Node.js and providing native support for TypeScript, there seem to be numerous examples and lessons to be learned from their advancements.

Since it hasn't been mentioned in this thread, I would like to inquire whether this proposal could enable the distribution of npm packages exclusively written in TypeScript. Such a capability would significantly enhance the user experience when working with Node.js and TypeScript, eliminating the need to juggle between .d.ts and .js files.

GeoffreyBooth commented 11 months ago

What is the latest update on this?

We’re exploring this space via https://github.com/nodejs/node/pull/49704 (cc @JakobJingleheimer). One idea we’ve been considering lately is that Node could possibly provide some kind of blessed or automated way to install plugins, one of which could be TypeScript. Part of the problem is that TypeScript itself doesn’t provide Node integration, and there are many third-party plugins for that purpose, and we would want to avoid picking a winner. There’s also no one correct way to set up TypeScript; it depends on your build tool and its configuration, your app and its framework and those expectations, and so on.

You might want to open an issue with the TypeScript team, if there isn’t one there already. They’re the ones who have the power to add official support for runtime transpilation and/or type-checking, and/or provide a blessed/recommended tsconfig.json for Node projects. TypeScript itself is somewhat coupled to Node, in that the extensionless resolution mimics Node’s CommonJS resolution and the conversion of ES modules to CommonJS modules was created to address former limitations of Node; the TypeScript team is perhaps the best source for how they advise integrating their tool with Node.