search

microsoft / TypeScript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
https://www.typescriptlang.org
Apache License 2.0
101.13k stars 12.5k forks source link

bug(esm): TypeScript is not an ECMAScript superset post-ES2015 #50501

Closed ctjlewis closed 5 months ago

ctjlewis commented 2 years ago

Bug Report

This bug report will show that TypeScript is no longer an ECMAScript subset as of ES2015 due to its refusal to support import specifier rewrites for contrived and misplaced reasons.

Preface

This issue is related to previously reported issues, but expands on them and clarifies the scope of the problem.

This team, mostly Andrew and Ryan, have repeatedly shut down these issues in the past. I ask that this issue not be closed on a kneejerk reaction because I intend to speak with others at Microsoft about it.

TS is designed to be an ES superset

TypeScript doubtless has many design goals, and I understand how hard it is to balance all of them. However, undeniably, its core design goal - what TypeScript exists to do - is stated in this repository's description and also throughout the TS docs:

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

We should all agree that TS, first and foremost, is supposed to be an ECMAScript superset. There is no amount of mental gymnastics that gets us out of this. We will proceed with this in mind as we cover the scope of this issue and some of the technical excuses that have been offered to justify not addressing the problem.

What is a superset?

Originally a set theory concept, in terms of language it was best stated by @sepp2k in this StackOverflow answer:

Syntactically a language A is a subset of a language B if every program that is valid in language A is also valid in language B. Semantically it is a subset if it is a syntactic subset and each valid A program also exhibits the same behavior in language B.

To formalize this, Sebastian offers the following definition:

If P_A is the set of all valid programs in language A and P_B is the set of all valid programs in language B, then language A is a syntactic subset of language B exactly if P_A is a subset of P_B.

We will not need to address the semantic subset issue as TS fails the first test for arbitrarily many input programs1, which we will get into.

For convenience, and to emphasize the point that our subset programs are actually emitted JS versions of the input TS program, we will refer to a program written in the superset language as "program T" and the corresponding output in the subset language as "program T'".

Additionally, because TS output depends on your compiler configuration, we will consider TypeScript to pass the semantic test if there is some configuration for which you can transform a valid T into T'.

1 This is something this team is already well aware of, and have simply chosen to ignore. In my judgment, the only possible explanation for violating the core purpose of this technology for such a large portion of its lifecycle is organizational deficiency, and not any practical technical reason.

Is TypeScript a JS superset?

Sometimes. Let's walk through some examples (with approximated and simplified, not literal, output):

  1. ✅ Program T

    const a: string = "hello world!"
console.log(a);

    ✅ Program T'

    const a = "hello world!"
console.log(a);

    The input program is valid and the output program is valid. The runtime behavior of T and T' are identical. This example passes both the syntactic and semantic superset tests.

  2. ✅ Program T (using relative extensionless TS import)

    import { a } from "./a"
console.log(a);

    ❌ Program T' (TS-style imports left untransformed)

    import { a } from "./a"
console.log(a);

    The input program is valid, but there is no configuration for which you will get a valid ECMAScript module output. The output program T' will throw according to the ES spec. This example fails the syntactic and semantic superset tests. Our perfectly valid TypeScript program cannot be transformed by the compiler into a valid ECMAScript program, and this is easily shown.

    While we can construct a different input program for which we will receive valid output (import { a } from "./a.js"), making that suggestion is proof positive that TS is no longer an ES superset, and there are infinitely many valid input programs which will not emit valid output. Nothing about this is subjective or within the realm of personal judgment.

Bearing this in mind, the last version of ES for which TypeScript was a superset would be ES5 (2009), before the introduction of ES modules.

Why does this matter?

It matters because our industry has struggled to adopt ESM for a variety of reasons, largely related to interop with downstream CJS consumers and also integration with TypeScript. It should be fairly obvious the damage that is done to the ecosystem when the TypeScript compiler very literally cannot generate valid ESM programs without special syntax.

The idea of having to use a special syntax to achieve a correct output program is inherently antithetical to the "TS is an ES superset" principle on which TypeScript is based. It takes a severe amount of technical dysfunction to turn a blind eye to this for years and years, and a serious amount of contempt for users and software quality to continually shut down user feedback on this issue.

Inadequacy of existing solutions and team response to user feedback

I have clarified several times that I am chalking this up to organizational dysfunction and attempting to kick it up the chain myself, as many many many users have raised this issue in isolation only to be shut down and/or directly gaslit by members of this team.

Let's go over some of these issues. I will start with the closest suggestion to a fix currently under consideration, and then go through the rest in chronological order.

(!) 2020: allow voluntary .ts suffix for import paths (#37582) (by @timreichen)

I will start with this suggestion as it comes the closest to fixing the problem, while also offering a two-birds-one-stone benefit with regard to Deno, Bun, etc. support:

let import a from "path/to/a.ts" behave the same as import a from "path/to/a"

As specified, this partially solves our problem. Because import "./a.ts" could only ever refer to a TypeScript program, you cannot conceive of any input .ts import statement which will not correctly build to a .js import and execute equivalently at runtime.

However:

  1. Because it is apparently "core design goal" that the imports never get rewritten, the most important part of this suggestion cannot be honored.

  2. (Very important) This demands you rewrite your existing, valid import "./a" statements to ./a.ts, and your valid ./a imports still do not build to valid ES import statements.

Thus, however we might choose to emit the statements from input, we must either:

  1. Resolve extensionless relative specifiers which are valid TS; OR
  2. Deprecate extensionless relative specifiers in TypeScript, and make them invalid TS, to bring T and T' to parity.

The current approach by this team has literally just been to do nothing and deny that there is a conflict here. This conflict and its relationship to TS as an ES superset is the core focus of this issue.

2017: TypeScript and (#13422) (by @cyrilletuzi)

Because the problem is, again, that TypeScript ESM input programs cannot build to valid output, naturally <script type="module"> goes right out the window.

Same pattern: User opens thread, years of comments and confusion, denialism from team members, summarily closed.

2017: Provide a way to add the '.js' file extension to the end of module specifiers (#16577) (by @quantuminformation)

This issue looks like the others we will see: A user raising the issue that their valid TypeScript program cannot compile to a valid ES program, a large debate between users and various TS maintainers, and eventually the issue being closed, locked, and buried.

As usual, users point out that TS will act like their imports are valid while refusing to resolve them at compile-time, highlighting the tradeoff I explained. Relative extensionless specifiers must either be allowed and transformed, OR disallowed entirely. Doing both only violates the TS superset principle and confuses users because they depend on their output programs being isomorphic with their inputs.

Also, to generalize this issue a bit, I don't think it's actually about adding a .js extension, but resolving the module specifier to an actual path, whatever the extension is.

This comment has 50 likes. The parent issue has nearly 300. Minimizing comments from the team were met with a negative reaction as usual.

2020: Compiled JavaScript import is missing file extension (#40878) (by @jameshfisher)

This is an interaction, and a restatement of the same problem, which is effectively the perfect example of not just the problem and the common sense solution to it, but also the community's investment in this being solved, and the textbook denialism by maintainers. The bot response announcing the closure as "working as intended" has 65 downvotes.

Mr. Fisher raises the issue that a program containing valid TypeScript imports will not emit a valid corresponding ECMAScript program. As we will see, this issue is met with approval and enthusiasm by people who found it trying to understand why their programs will not build (after several hours, they will presumably put two and two together and realize it's just not possible). OP is then promptly sandbagged by members of this team and the issue is erroneously closed.

After the issue is posted, Ryan quickly responds to this post by minimizing OP's complaint and telling him to kick rocks:

TypeScript doesn't modify import paths as part of compilation - you should always write the path you want to appear in the emitted JS, and configure the project as necessary to make those paths resolve the way you want during compilation

Again, as we've covered, this is not how supersets work. Simply telling someone to rewrite every already-valid TS import statement into a different one which will compile should seem absurd on its face given what we've gone over. This comment received about 20 downvotes.

Mr. Fisher responds noting that he was shocked to find referring to TS source files with .js (an extremely confusing and hacky solution that is unbecoming of even a temporary workaround, let alone a permanent solution) actually does resolve his problem. He then lays out a few of the many reasons why this is confusing and hacky. His comment received 50 positive reactions.

Ryan responds with another heavily-downvoted dismissal of this issue:

TS never takes an existing valid JS construct and emits something different, so this is just the default behavior for all JS you could write. Module resolution tries to make all of this work "like you would expect"

You would expect a valid TS import to become a valid JS import because it's supposed to be a superset. It is not a discussion or an issue of perspective; TS claims to be a superset, if it is, the valid imports would remain valid at runtime. Yet they do not, and we have highly experienced and talented engineers effectively lying about what is and is not expected runtime behavior.

@RevealedFrom clarifies this at further length, though it should be pretty obvious already why there are several problems with this, least of all being the superset issue:

@RyanCavanaugh Writing "./dep.js" doesn't sound logical. The file dep.js does not exist in the Typescript universe. This approach requires the coder to know the exact complied output and be fully aware of the compiled environment. It's like having to know the CIL outputted and modify it here and there in order to code in C# successfully. Isn't the whole idea of Typescript to abstract away Javascript?

import { foo } from "./dep" is legitimate Typescript, and it provides the information for Typescript to resolve all that is needed to type check and make the code compile successfully. So, the compiled output should work. Typescript should not be generating syntactically incorrect Javascript.

IMHO, this issue should be a bug.

Ultimately, it is a bug, and we know it is a bug for the reasons we've covered. Ryan responds again with a minimizing and heavily-downvoted statement I would characterize as dishonest at best, and an outright lazy lie at worst:

The whole idea of TypeScript is to add static types on top of JavaScript, not to be a higher-level language that builds to JS.

We have enough background to know this is wrong, though we are starting to get insight into the executive dysfunction that is causing this issue. Core maintainers are making up what the purpose of this technology is as they go along, apparently now privately rejecting that TS is or ever was meant to be a JS superset.

A user in this thread (@Gin-Quin) had to actually implement a bundler to work around this, something I also had to do:

I will add that I indeed created a library called tsc-esm as a workaround for this bug, and even though it works quite fine every time I use it (ie 100% of the times I need to create a library written in Typescript) I feel I should not have to patch the output of the Typescript compiler like I do.

And another user (@djfm) also:

@silassare I think probably a thousand person, me included, have written similar scripts. This is just absurd.

And the last comment from this thread I will cover, and emphasize, is this one by Mr Felber (@PatrickFelber):

This whole discussion is absurd. Valid typescript obviously transpiles to invalid ES6 code. I do not understand in which world this can be classified as "works as intended" behavior. Adding ".js" to import statements which actually refer to ".ts" files as workaround is too wild for me. For my current project I have free choice, therefore I'm gonna switch over to CommonJS target module code. CommonJS works without file extensions. In case one can live with CommonJS target module, this is an option to get around this problem.

The only people who claim to not see the problem are members of this team. Note that this user's only choice was to give up and stay on CJS - this is what I am referring to when I say that this is likely the most major blockade to ESM adoption under Why does this matter?.

2020: TypeScript cannot emit valid ES modules due to file extension issue (#42151) (by me)

This issue was opened by me to clarify that the compiler cannot emit valid ES modules. It received positive feedback from users, but was naturally accompanied by denialism from Ryan and eventually closed and locked. A comment by @richardkazuomiller points out the ridiculousness of the justification offered for not fixing this issue ("we can't rewrite import specifiers"):

Exactly, could someone explain like I'm 5 why it's OK for TypeScript to convert import ... from './file' to const ... = require('./file') changing ./file to ./file.js is not OK?

2021: One year later: TypeScript still cannot emit valid ES modules (#47270) (by me)

This is a rephrasing of the original issue, posted so as to remind this team that this problem has not been resolved. The single response from Ryan was amusing given the context of this issue:

We've explained the rationale behind not modifying import paths on emit many times, and don't see value in rehashing that discussion since it relates to a core design goal. [...] Please file something concrete in the future.

You must decide to prioritize a TypeScript "core design goal", but you can only choose one:

  1. Never rewriting import specifiers, even though the compiler does this for regular import statements
  2. Ensuring TS is an ES superset like it says it is throughout its documentation

Apparently this is not an obvious answer to TS maintainers, though to users it apparently is. Furthermore this issue is an attempt to follow-up explicitly on the second half of Ryan's request to file something concrete going forward, though I imagine this issue will also be summarily buried as this team appears to have only one tool in its toolbox, namely denial for the sake of convenience.

2020: Appending .js extension on relative import statements during Typescript compilation (ES6 modules) (StackOverflow)

Same situation we've seen: User has a valid input program, can't get a valid output program. The answerer prudently notes:

If the compiler simply generated extension-less output files it would also solve the issue. But, would that also somehow violate the design principle regarding URI rewrites? Certainly, in that case there could exist other design principles to defend the position! But wouldn't such stubbornness only help to further validate the adamancy or ignorance of the TS team on this issue?

2022: "module": "node16" should support extension rewriting (#49083) (by @arendjr)

This is the most recent issue that covers this problem. Mr. van Beelen is certainly a lot more generous and forgiving with his wording than I am, though he is equally rewarded with stark denialism from Ryan.

We don't rewrite target-legal JavaScript code to be different JavaScript code from what it started with, no matter the configuration.

We won't even get into the fact that import "./a" is not an ES2015 "target-legal" import statement, as it's just an egregiously incompetent claim. It received about 60 downvotes.

Mr. van Beelen left several more long, thoughtful comments, generating over 100 positive responses, explaining much of the same things that I have covered here, that valid TS programs should become valid JS programs, though he did so with much more forgiving phrasing. I recommend everyone read his comments, and I found them very considerate - likely impractically so.

This is another thread that contains hundreds of comments, hundreds of positive reactions for the sensible position taken by users, and hundreds of negative reactions for this team's excuses. It is impossible to read it all, but it is worth skimming over and noticing that it, like every other issue raised regarding TS's inability to build valid output programs, was summarily closed for the sake of maintainer sanity to the detriment of several million codebases relying on TS.

2022: Add Compileroption to add Extensions to relative imports (#47436) (by @jogibear9988)

A PR to add non-breaking, opt-in compilerOption for adding .js extension to extensionless relative TS imports, which received about 30 positive reactions. Of course, it was summarily closed without a real review. No amount of attempts to contribute by the community have been well-received by this team.

🔎 Search Terms

🕗 Version & Regression Information

TypeScript was initially released in 2012, and it has not been an ES superset since the release of ES2015, which require imports to have file extensions if they are not named modules.

⏯ Playground Link

Cannot repro in playground, requires multiple entry points. Minimum repro: https://replit.com/@ctjlewis/ts-esm-superset

Throws ERR_MODULE_NOT_FOUND because ./a is not a valid ES import specifier.

💻 Code

N/A, see repro and notes in this issue.

🙁 Actual behavior

The valid TS-ESM program T builds to an invalid program T'.

🙂 Expected behavior

The valid TS-ESM program T should build to a valid program T'. Valid TS imports should always compile to valid ES imports for the given target.

fatcerberus commented 2 years ago

I stopped reading here:

The output program T' will throw according to the ES spec.

The ECMAScript specification says nothing about the meaning of module specifiers. Module resolution is the host’s responsibility and is therefore implementation-defined. If the TS compiler can resolve an import but your runtime can’t, that just means your project is incorrectly configured for the target runtime environment and doesn’t say anything about ES compliance.

If you want to argue this behavior is suboptimal, that’s fine, but you can’t use spec compliance as a bludgeon here.

Josh-Cena commented 2 years ago

Sigh so many people thinking that explicit extensions and full paths are ES requirements is really Node's fault. They could well keep the CJS resolution mode and make everyone happy. (Same goes for JSON modules—soon people will think assert { type: "json" } is required for JSON modules, just because Node is paranoid about aligning with browsers these days.)

Also, if you use ts-node, you could well make your code run with .js extension imports via the resolver—it's host-defined behavior at the end of the day.

ctjlewis commented 2 years ago

Module resolution is the host’s responsibility and is therefore implementation-defined

This is technically true (though in practice, pedantic).

The ECMAScript specification says nothing about the meaning of module specifiers.

This is not true:

Multiple different referencingScriptOrModule, specifier pairs may map to the same Module Record instance. The actual mapping semantic is host-defined but typically a normalization process is applied to specifier as part of the mapping process. A typical normalization process would include actions such as alphabetic case folding and expansion of relative and abbreviated path specifiers.

ECMAScript 2023 § 16.2.1.7

We can go round and round on this if you want, but whether you're in the browser, you're in Node, or any other major host of note, in pure ES, import "./file.js" will be valid output and import "./file" will not be.

Josh-Cena commented 2 years ago

"Include actions such as" is extremely handwavy. It's neither normative nor actually specifying any algorithmic steps. It says nothing about import "./file" being invalid or import "./file.js" being valid.

fatcerberus commented 2 years ago

An example of “typical” use does not constitute normative text.

ctjlewis commented 2 years ago

Stockholm Syndrome ITT.

ctjlewis commented 2 years ago

We've had thousands of people confused and upset over this for the last 5 years because their valid input programs end up failing to generate valid output.

On the basis of that alone you're wasting your time, you're not going to convince me that doing nothing is the appropriate solution, thanks for the comments though.

fatcerberus commented 2 years ago

On the basis of that alone you're wasting your time, you're not going to convince me that doing nothing is the appropriate solution, thanks for the comments though.

Like I said, if you want to argue the current behavior is suboptimal, that's fine and I'd even be likely to agree with you. But trying to using the specification as a bludgeon is dishonest here.

ctjlewis commented 2 years ago

Like I said, if you want to argue the current behavior is suboptimal, that's fine and I'd even be likely to agree with you. But trying to using the specification as a bludgeon is dishonest here.

OK, fine, I take your point with the limited note that it doesn't say nothing about it, and also point to implementations in practice. There's enough dishonesty in this whole issue to go around. But yes I will grant you that the spec does not explicitly specify this - we'd probably agree that it could and should.

Though, to be clear, import "./a" is not even target-legal for Node target. So I mostly dispute that there's a substantive issue with my statement.

quantuminformation commented 2 years ago

in my most recent vanilla browser module experiments, they are written in what language do you imagine?

i’ll give you a tip it begins with j and ends with s

On Mon, 29 Aug 2022 at 04:42, C. Lewis @.***> wrote:

Like I said, if you want to argue the current behavior is suboptimal, that's fine and I'd even be likely to agree with you. But trying to using the specification as a bludgeon is dishonest here.

OK, fine, I take your point with the limited note that it doesn't say nothing about it, and also point to implementations in practice. There's enough dishonesty in this whole issue to go around. But yes I will grant you that the spec does not explicitly specify this - we'd probably agree that it could and should.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/TypeScript/issues/50501#issuecomment-1229683923, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABU35SXLH6UQ4F65IEAC5LV3QPQRANCNFSM574CX7OQ . You are receiving this because you were mentioned.Message ID: @.***>

-- Kind regards Nikos Katsikanis https://nikoskatsikanis.com Tech Innovation | Podcast Host | Linktr.ee https://linktr.ee/nikosshow

jcalz commented 2 years ago

I'm probably going to regret wading into this, but I wouldn't frame this issue in terms of language supersets, if the goal is to change how import statements are emitted.

"TS is a superset of JS" means that every valid JS program is also a valid TS program, in a particular sense. It basically just means that a valid JS program should be emitted as-is (or possibly downleveled). It's quite a weak statement. The only violations of this I'm aware of are things like parsing edge cases as in #33639.

It doesn't imply that every valid TS program will be emitted as a valid JS program, although obviously such behavior is important. It might not be desirable to emit some line of TS as-is to JS, but doing so doesn't violate "TS is a superset of JS".

magic-akari commented 2 years ago

Does anyone expect to get an error report instead of silent output?

try "moduleResolution": "Node16", you will get the TS2835 error.

// foo.ts
export const foo = 42;

// bar.ts
import { foo } from "./foo";
//       ^^^
//        |----[TS2835] Relative import paths need explicit file extensions in EcmaScript imports when '--moduleResolution' is 'node16' or 'nodenext'. Did you mean './foo.js'?
SerLizar commented 2 years ago

Like I said, if you want to argue the current behavior is suboptimal, that's fine and I'd even be likely to agree with you. But trying to using the specification as a bludgeon is dishonest here.

Let's forget about the specification then, given that those using TS dislike the current behaviour, and you are likely to accept it is suboptimal. How about we finally change it!

I really don't care if I have to use a config file and CLI argument at the same time to make it work.

It doesn't imply that every valid TS program will be emitted as a valid JS program, although obviously such behavior is important.

If you describe TS as a superset of JS, this is correct (in TS is JS with extra stuff, some of that could make it incompatible). However, the problem is that:

TypeScript adds optional types to JavaScript that support tools for large-scale JavaScript applications for any browser, for any host, on any OS. TypeScript compiles to readable, standards-based JavaScript.

If it should work in any browser, changing './foo' to './foo.js' is desirable. Unless you want to take the battle to the browsers, changing the current behaviour of TS looks like the way to go.

avin-kavish commented 2 years ago

The whole idea of TypeScript is to add static types on top of JavaScript, not to be a higher-level language that builds to JS.

Not a fan of this backtracking either, 5 years ago this was not the rhetoric that brought me onto TS. TS offered much more, it provided design constructs that were lacking in js and also more expressive syntax (which eventually got added to js). But mostly, the design constructs of access modifiers, statics, interfaces and classes made it possible to design large systems without breaking a sweat. Maybe the differences in the ts implementation and eventual es standard is now deterring the team from adding new features to the language? Case in point, # vs private. legacy decorators. different field initializers. So it looks like the goal post was (silently) moved? Maybe it is also to appeal to the masses of javascripters? I don't believe this is entirely good. TS used to show what JS could be. Now only babel can do that, but new syntax that compiles through babel is not compatible with the ts language server. So users have to chose between new features with babel on js vs ts with current features. Also babel is limited to whatever is proposed to Tc39, which is a very small subset of what a language can be. Just look to kotlin, swift and even recent versions of c# for inspiration. Languages have evolved a lot in the past few years and TS hasn't moved much in terms of non-type features. But type features dissapear at compile time and don't add any value at runtime. So I think excessive type-safety is overrated, moderate type-safety is good. As one wise medium user once said to me, "a sound type system does not maketh a language". TS is not forward looking anymore, it's only making what works in js typesafe. I think that was important for full adoption. But I don't think it should be the only goal. I think it should provide a superset of js.

Regarding this particular mechanic, I mean we have a flag that says "allowESModuleInterop" that transforms imports. So I don't see why adding another flag, "suffixESImportsOrWhatever" that transforms imports in a slightly different way would be too much of a stretch to imagine. Whether it can accurately do that though, while remapping things like index imports, I don't know. I think this whole add-.js-at-the-end fiasco is completely unnecessary trouble. Why did we go from nice imports to ugly imports again? Sounds like a step back for mankind.

I'm solutions-oriented though, most times the fastest path to a solution is to write your own code, like below.

Solution

Problem: I want TypeScript to automatically add .js extensions to my imports.

In cases like this I use ttypescript, it basically plugs into the compiler api and lets you run arbitary code as it visits nodes in your AST. Scroll down the readme and you will find a plugin that does exactly what you want, i.e. add .js extensions to your relative imports.

Also, I wrote this hacky but working script to do a one time migration of my code base.

import glob from 'glob'
import { readFileSync, writeFileSync } from 'fs'

glob.glob('src/**/*.ts', (err, matches) =>
  matches.forEach(match =>
    writeFileSync(
      match,
      readFileSync(match, { encoding: 'utf-8' })
        .split('\n')
        .map(l =>
          l.startsWith('import') && l.includes("'.") && !l.endsWith(".js'")
            ? l.replace(/'$/, ".js'")
            : l,
        )
        .join('\n'),
      { encoding: 'utf-8' },
    ),
  ),
)

(js is so expressive these days, you can do all that in a single statement.)

Also, shoutout to tst-reflect which through ttypescript gives much better reflection than anything else we have "officially" atm.

ctjlewis commented 2 years ago

So I don't see why adding another flag, "suffixESImportsOrWhatever" that transforms imports in a slightly different way would be too much of a stretch to imagine.

Exactly correct. There is no obstacle to adding non-breaking, opt-in support for this.

Noted in the issue, someone actually added this flag and sent in a PR, but it was summarily closed. It is a striking example of organizational deficiency in software. I suspect a single decision-maker or a small group of them are the only obstacles to un-breaking the compiler in this respect.

ctjlewis commented 2 years ago

@jcalz And it remains a superset if a mapping from the superset to a subset is not a valid subset?

Really, that's where we're going with this to justify emitting broken programs?

jogibear9988 commented 2 years ago

@ctjlewis i added an option in my pull req: https://github.com/microsoft/TypeScript/pull/47436

ctjlewis commented 2 years ago

@ctjlewis i added an option in my pull req: https://github.com/microsoft/TypeScript/pull/47436

I know, I mentioned and linked it - thanks for sending that in, sorry it got buried. That was a good contribution and I appreciate it.

jogibear9988 commented 2 years ago

and it could also be solved with custom transformers, but they are not public. don't know why they exist at all when they are not exposed

ctjlewis commented 2 years ago

Worth highlighting Ryan's comment from the linked response to that PR too:

Maybe we have failed, I don't know, but we're going to keep going on this path either way.

Awesome. We're plugging our ears and yelling "imports are pure JS! imports are pure JS!" even though we can write TypeScript-specific, non-target-legal ./relative imports which are not pure JS and do need to be transformed.

Yes, you are failing, but the beauty of software is we can always fix it. I'm still holding my breath.

jcalz commented 2 years ago

@jcalz And it remains a superset if a mapping from the superset to a subset is not a valid subset?

Yes.

Really, that's where we're going with this to justify emitting broken programs?

I’m not really trying to justify anything. I’m pointing out that this issue isn’t about what the title is claiming to be about. And that talk of supersets is a distraction from your actual concern here.

By the way, I’m not a member of the TS team or anything, so I certainly wouldn’t be in a position to authoritatively justify TS emitting broken programs, working programs, or mp3s of flugelhorn music.

nicolo-ribaudo commented 2 years ago

You know that import { foo } from "./foo" works perfectly in the browser, right? Your server just has to respond to GET /path/to/foo with a file that has text/javascript mime type. This is identical to how it has to respond to GET /path/to/foo.js with a file that has text/javascript mime type when using import { foo } from "./foo.js".

richardkazuomiller commented 2 years ago

I shared my opinion about how I would have liked TypeScript ESM imports to work on other issues and I didn't get everything I wanted, but what we have now works and many projects are finally able to migrate to ESM so I'm happy nonetheless.

richardkazuomiller commented 2 years ago

Actually, my original complaint was that TypeScript didn't throw any errors or show warnings if you wrote imports the CJS way and compiled to ESM. Now it does, so I kind of did get everything that I wanted regarding things that are most important in my opinion.

egasimus commented 2 years ago

Okay, let's say I have three files, foo.ts, bar.ts, and baz.ts.

// foo.ts
import { baz } from './bar.js'
// bar.ts
export { baz } from './baz.js'
// baz.ts
export function baz () { /* ... */ }

Let's also say that I use a framework which compiles and loads TypeScript on demand, without ever emitting JavaScript. So, the files bar.js and baz.js never exist. In other words, for things to work, I need to import from non-existent file.

I understand that Node is somewhat at fault here for nerfing ESM/CJS backwards compatibility in a way that looks very much the worst of both worlds. Why doesn't TypeScript:

instead of:

Now let's say I have the above three files, but I import without extensions (import { baz } from './bar', etc). The framework loads everything correctly, watches files, reloads on edit, even does nifty in-place hot reloading, etc.

Now let's say I want to publish part of my repo as a library in the form of a portable ES module on NPM. I run TSC, which emits foo.js, bar.js, baz.js. Now these files exist but contain the (invalid according to Node) extensionless imports. So I diligently add the weird .js extensions in the original TS source and what happens?

:clap: Reloading :clap: stops :clap:

Now I have to re-run TSC on every change, where I previously didn't, because bar.ts and baz.ts have fallen off the dependency graph and the dev server only sees the .js variants which will go out of date the first time I change the original sources.

Again, please point out the person to whom this affront makes sense, and let them explain how this is not an arbitrary speed bump introduced into the otherwise generally sane modularity and packaging workflow that the JS community had largely settled on before you bought your pseudo-superset a place the ecosystem. I agree that what the spec says is beside the point, because TypeScript is not only non-compliant, it simply generates code that doesn't work and, to make it work, requires the programmer to write code that doesn't make sense.

And it requires developers to wrap their heads around numerous implementation details that simply did not exist previously. In my experience, the average JS dev I've met already has a hard time wrapping their head around how Promises and async/await are the same thing, you're just making this harder on people and businesses trying to solve problems by using your platform, which of course is widely lauded as the next best thing since sliced bread. So you can't even choose not to use it. Congrats, you've "won".

Yall really overdid it with this one, folks. This particular abstraction is so leaky that if it was a ship it'd be sinking. I'd like to say that it ruined my day - only it's not just one day and it's not just mine. You know what they say, "don't look for malice where things can be explained by incompetence". To which I ask - what is malice, in our industry, other than organized incompetence?

ctjlewis commented 2 years ago

@egasimus Go off king. Also, sadly, you'll find several people ITT willing to suck up and make excuses for this problem that could've been resolved years ago. Stockholm Syndrome is a hell of a drug.

ctjlewis commented 2 years ago

You know that import { foo } from "./foo" works perfectly in the browser, right? Your server just has to respond to GET /path/to/foo with a file that has text/javascript mime type. This is identical to how it has to respond to GET /path/to/foo.js with a file that has text/javascript mime type when using import { foo } from "./foo.js".

OK, so you need to customize the behavior of the server rather than serve a simple filesystem. Proved the point for me.

iT wOrKs pErFecTlY iN tHe bRowSer (with my super special server configuration) - lmao.

@jcalz And it remains a superset if a mapping from the superset to a subset is not a valid subset?

Yes.

Does it remain a superset that can map to the subset of it doesn't map to the subset, since TS literally is designed to meet this definition as we have already covered and as is mentioned in all of their documentation?

We can ad hoc it as much as we want, it doesn't meet its technical goals in an ESM context. No amount of cope will change this.

staplespeter commented 1 year ago

So, I encountered this problem today.

I first used JScript (remember IE ran it's own flavour) back pre-2000 when browsers started supporting scripting and I thought it was the bees knees. It was only my second language so couldn't judge. 20 years later I've returned to this side of web dev to see what the fuss is about. Being largely open source there are clear inefficiencies, some dodgy code and general workarounds needed for issues, but by and large everything works as intended and described. There are clearly lots of dedicated teams of highly skilled individuals both creating and maintaining, and also a strong and helpful community of developers assisting on forums. I'm happy to be a part of this.

When starting to learn this ecosystem I want to be at the forefront of the tech. I want to use what's good for the future so I'm writing TS and using ESM. Natch, bro. At least that's what all the docs and tutorials tell me to do.

I've written a MySQL datasource, recordsets, DAOs and a model, an API and client React code. As I'm new to this ecosystem I've been carefully reading the docs on each tech and lib, following tutorials, writing unit and integration tests as I go.

So I've overcome a fair few issues to get to the point where I go to deploy my server (perhaps i should have tried sooner but... tests!), and I get: Error [ERR_MODULE_NOT_FOUND]: Cannot find module 'X' imported from Y. I look at my build .js files and see there is no .js extension in the import so I add it. The error then reports for the next dependency.

Surely that can't be it??? Surely tsc knows I'm writing TS and would add the .js extension to the output, because it's creating JS??? I mean I've followed every tutorial to the letter, referenced www.typescriptlang.org docs dozens of times and not once has anything been said about tsc not producing valid ESM output. I know ESM support in Node is experimental, but then so should tsc and thus allow for it's behaviour. And I've been encouraged to use ESM as it's the future, so why is this failing on me now??? (I will add that since I started on my learning the https://www.typescriptlang.org/docs/handbook/2/modules.html page has been updated to specify that .js extensions are used on ESM imports, which is something at least).

Of course I find #42151 and #16577.

I understand that ESM does not require an extension and that browsers can request ESM imports from the server and that an output .js file with a module reference ./X with no file extension can refer to ./X.js or ./X/index.js. This argument has been made as a reason tsc can't add a .js extension.

This reason is clearly wrong. tsc knows I am transpiling a .ts file. Presumably it has built a dependency graph so it knows that all my dependencies exist in other .ts files in known locations. It could ensure all files are referenced by name e.g. ./X -> ./X.js or ./X -> ./X/index.js. It can certainly add an extension. And my code isn't going to hit the browser and then I'm going to sneakily rename or move my file so ./X no longer means ./X.js but ./X/index.js.

And I'm not even building for the browser (client code uses Parcel), so the expected issue CANNOT even arise. If only I could FLAG (hint hint) to tsc somehow that it needn't worry about paths and i just want it to output .js module references. But this is moot, as tsc already knows the full location and type of all dependencies.

There is another argument used that valid JS must not be altered during transpilation. Okay, so if my import was explicitly ./X.js then great. I'm writing TS with extra language features that in some cases supercede JS for best practice. I'm also using a TS transpiler so I expect it's features to also supercede, or override, those related to JS. Specifically, if i am referencing a module written in TS I expect this to be handled accordingly i.e. for tsc to produce the correct output in JS for ESM. Again, this is because tsc knows EXACTLY what it is transpiling. There is no technical reason why it cannot do this.

I will also point out that on the https://www.typescriptlang.org/ homepage the following is immediately visible: TypeScript code converts to JavaScript, which runs anywhere JavaScript runs: In a browser, on Node.js or Deno and in your apps It doesn't convert my TypeScript ESM import to working JavaScript. In order for this to occur I have to add JavaScript to my TypeScript file i.e. import A from './X.js'. If tsc added the .js for me then the above claim would be true.

Perhaps Node.js should resolve it's dependencies better. It sure could check for ./X.js and ./X/index.js. However the TS devs know they are targeting a limited number of platforms - at this point the vast majority are ESM compliant browsers or Node.js. If the TS devs want to pass the buck to Node.js the least they could do, in support of their thousands of customers asking for a solution, would be to liaise with the Node.js team and agree a solution between themselves. In the spirit of this I have added a feature request to Node.js repo https://github.com/nodejs/node/issues/46006.

My speil at the start was to highlight that web tech has been constantly evolving, pushing forward relentlessly and encouraging people to use newer and better (?) technologies, whilst capturing the best practice as standards for future conformity, stability and efficiency. It's not an easy road but THAT is the right ideology.

The solutions suggested, and implemented, in TypeScript effectively place the burden on the developer to ensure that they have the '.js' extension added to all ESM imports. This is opposite of what useful programs do. Software tools are there to make the life of the developer easier. Currently TypeScript and tsc make life harder for the developer on this issue.

Ideally the team here would be supporting the drive for standards, for ESM, for adoption of TypeScript over JavaScript, making it easier for developers, especially new ones, to create apps without headaches and workarounds, without needless ideological and bullheaded limitations. Ideally I'd like to see JavaScript disappear and TypeScript to be the language.

What I've got instead is frustration and disappointment. It really makes me not want to use TypeScript and Node.js. I can only imagine how insanely confusing and frustrating it must be to a new dev.

arendjr commented 1 year ago

@staplespeter I understand your frustration, but I think if you’re starting out fresh with a new project, it’s better to start with Deno rather than TSC+Node. Not only does it have better security and interoperability with web standards, it also takes care of the TypeScript experience so you don’t have to deal with issues such as these. Unfortunately, the TS devs themselves have shown themselves unreceptive to this kind of feedback.

staplespeter commented 1 year ago

@arendjr Thanks for the recommendation. Yes going to try Deno and use it from now on.

The response from Node devs was that this has been discussed before and they aren't going to do anything with paths with no extensions. Their docs clearly state that ESM relative import paths must have a .js or .mjs extension. I think they could resolve these extensionless paths without difficulty but here we have it.

jogibear9988 commented 1 year ago

I don't think it's a node issue, I still think it should be solved in TS. Cause Browsers will never work without extensions. I alteady created a pull with a possible fix, but as everybody know, the team here does not want it.

staplespeter commented 1 year ago

I agree, and the Node devs had the same opinion also. They did recommend some solutions (which I wasn't terribly appreciative of), which I may try to implement if Deno doesn't live up to expectations

ekwoka commented 1 year ago

Seems wild they've done this now after allowing it to work before...

So now we have a situation where

packages without a types field, because they use exports won't have types resolved in node moduleresolution

and relative imports require a .js filename in nodenext and node16 moduleresolution.

Why change what wasn't broken?

quantuminformation commented 1 year ago

I still can't use ts for my vanilla stuff 6 years later >< maybe we get native types in js though

hence js away > https://www.youtube.com/watch?v=rURpVmzBQtU

Aeolun commented 8 months ago

Well, I'm happy that after 8 years, at least we have an open issue to complain on. If nothing else, it'll make me feel better when I come back here again in 6 months.

ctjlewis commented 8 months ago

Well, I'm happy that after 8 years, at least we have an open issue to complain on. If nothing else, it'll make me feel better when I come back here again in 6 months.

Don't worry, I never let them hear the end of it. Thankfully, we have Bun now. Use that for TypeScript + ESM builds.

ekwoka commented 8 months ago

I think it's gotten better.

Configing to allow no extensions or .js or even .ts works well now even with exports, and even exports with custom conditions!

I do kind of disagree with the original post, that typescript being a superset does not require that all valid TS programs output 100% valid ES code after compilation. A superset means all valid JS programs are valid TS programs, not that valid TS programs always make valid JS programs.

ctjlewis commented 8 months ago

I do kind of disagree with the original post, that typescript being a superset does not require that all valid TS programs output 100% valid ES code after compilation. A superset means all valid JS programs are valid TS programs, not that valid TS programs always make valid JS programs.

It's a serious stretch but I think it still makes sense in the context of transpilation.

The phrasing is pretty obviously overkill, but the politics on this issue were crazy.

nurse-the-code commented 8 months ago

So I am a newcomer to all of this. If I am understanding this correctly, the NodeJS devs say it's a TypeScript problem, and the TypeScript devs say it's a NodeJS problem. And rest of us who use those platforms are caught in the middle.

https://stackoverflow.com/questions/75807785/why-do-i-need-to-include-js-extension-in-typescript-import-for-custom-module

What is the officially-recommended work around suggested by the TypeScript maintainers? I just want to be able to use ES6+ import statements and have it compile correctly into JavaScript.

RyanCavanaugh commented 8 months ago

What's the problem you're trying to solve?

arendjr commented 8 months ago

@nurse-the-code I can’t speak for the TypeScript maintainers, but I would suggest using either Deno or Bun, since they can evaluate TypeScript directly and don’t require compilation. You can use .ts extensions in your import statements with both.

If you do want to keep using NodeJS, you’re usually better off using a bundler such as Rollup or ESBuild, rather than using TSC for compilation. Bundled outputs are usually smaller and faster to load.

Basically, for most projects I would recommend using TSC only for type-checking and use something else to bundle or execute the code.

llllvvuu commented 8 months ago

IMO,

  1. "node will emit code that doesn't run on Node" is a real issue.
  2. "nodenext will refuse to emit code that doesn't run on Node, nor will it auto-fix your code" is a non-issue. I have forked an ESLint plugin (npm) to auto-fix missing extensions, although I have never actually needed it since auto-import already does the right thing.
  3. If 2 was indeed an issue, "bundler is fine with emitting code that doesn't run on Node, so you can have a separate step that fixes it" solves the issue.

I think 1. can be resolved by deprecating "moduleResolution": "node". All future es* targets should default to either "moduleResolution": "nodenext", or some variant of "nodenext" stripped of node_modules (https://github.com/microsoft/TypeScript/issues/37971), rather than

Classic if module is AMD, UMD, System, or ES6/ES2015; Matches if module is node16 or nodenext; Node otherwise.

Maybe there is also some documentation fix to explain points 1/2/3 better.

Now I have to re-run TSC on every change, where I previously didn't, because bar.ts and baz.ts have fallen off the dependency graph and the dev server only sees the .js variants which will go out of date the first time I change the original sources.

This is untrue. Go to definition et al go to the TypeScript source file. Bundlers in watch mode work the same way they always have.

RyanCavanaugh commented 8 months ago

The documentation already explicitly dis-recommends moduleResolution: "node"

nurse-the-code commented 8 months ago

Thanks for the responses!

The solution I went for was using the create vite vanilla TypeScript template. Under the hood it uses Rollup. In the process, I also discovered pnpm. Overall, I am pretty happy with my initial results.

I have a functioning starter project, and I can easily transpile the following TypeScript:

import Foo from "./foo.ts";

into functional JavaScript that does what I need it to do.

For those interested, here are links to info on Vite, Rollup, and pnpm.

I also checked out @arendjr's suggestions of Deno and Bun. They looked like interesting projects, but I was a bit nervous to move that far away from the npm/node ecosystem. Choosing 'vite', 'rollup' and 'pnpm' seemed like a more moderate choice.

ekwoka commented 8 months ago

Vite actually uses esbuild for the ts transformation, not rollup.

arendjr commented 8 months ago

Vite actually uses esbuild for the ts transformation, not rollup.

I think that’s a bit misleading, because they use rollup for module resolution, which is what we’re discussing here. For syntax transpilation it can use either SWC or ESBuild.

MiirzaBaig commented 5 months ago

Was it able to be resolved?

RyanCavanaugh commented 5 months ago

See https://github.com/microsoft/TypeScript/wiki/FAQ#module-specifier-rewriting

ctjlewis commented 5 months ago

Was it able to be resolved?

The solution is use Bun, and not depend on the compiler built by the TypeScript team.

guest271314 commented 5 months ago

We can go round and round on this if you want, but whether you're in the browser, you're in Node, or any other major host of note, in pure ES, import "./file.js" will be valid output and import "./file" will not be.

This might be of interest to you Intercepting and handling arbitrary static and dynamic Ecmascript import specifiers, protocols and file extensions.

We have WICG Import Maps now. We can do this https://github.com/guest271314/webbundle/blob/browser/index.html#L5-L18

  <script type="importmap">
    {
      "imports": {
        "Buffer": "https://gist.githubusercontent.com/guest271314/08b19ba88c98a465dd09bcd8a04606f6/raw/f7ae1e77fb146843455628042c8fa47aec2644eb/buffer-bun-bundle.js",
        "base32-encode": "https://esm.sh/base32-encode@2.0.0",
        "cborg": "https://esm.sh/cborg@1.10.2",
        "commander": "https://esm.sh/commander@4.1.1",
        "mime": "https://esm.sh/mime@2.6.0",
        "to-data-view": "https://esm.sh/to-data-view@2.0.0",
        "wbn": "https://esm.sh/wbn@0.0.9",
        "wbn-sign-webcrypto": "./wbn-sign.js",
        "zod": "https://esm.sh/zod@3.22.4"
      }
    }
  </script>

and this https://gist.github.com/guest271314/78372b8f3fabb1ecf95d492a028d10dd#the-code

<script type="importmap">
  {
    "imports": {
      "test/file": "./x.json",
      "test/sub/file": "./z.json"
   }
 }
</script>
<script type="module">
  import x from "test/file"
  with {
    type: "json"
  };
  import z from "test/sub/file"
  with {
    type: "json"
  };
  console.log(
    new TextDecoder().decode(new Uint8Array([x, z])), 
    import.meta.resolve("test/file"), 
    import.meta.resolve("test/sub/file")
  );
</script>

and this 

deno run --unstable-byonm --import-map=import-map.json -A index.js

The solution is use Bun, and not depend on the compiler built by the TypeScript team.

My solution is to indeed bundle all TypeScript source code I use in my programs, or want to test or break, or use in the browser or just read, to JavaScript with bun build or deno bundle while it's still around https://gist.github.com/guest271314/08b19ba88c98a465dd09bcd8a04606f6, https://gist.github.com/guest271314/1e8fab96bd40dc7711b43f5d7faf239e.

My broader solution to use multiple JavaScript engines and runtimes for development. I don't entertain a preference for any. They all have features that the other doesn't have, and implement things differntly. E.g., Bun.stdin.stream() behaves differently from Bun.file("/dev/stdin").stream(); Deno's dynamic import() is not really dynamic when using static string specifiers; Node.js doesn't support import maps, nor does Bun.

The semantics of what constitutes ECMAscript conformity or not is ultimately a politically hierarchical decision, it's not a democratic process.

If you use TypeScript, use TypeScript. There's only one, that I am aware of, even though

Typescript Specifications version #15711

There are about four people on the planet who can accurately update the spec and all of them are quite busy already.

If you are using JavaScript I would strongly suggest, if only for an experiment, if not a standard practice or policy, which is what I do, trying to run your source code using multiple JavaScript(/TypeScript) runtimes, constantly. There are few dozen, at least A list of JavaScript engines, runtimes, interpreters, to experiment with and develop with simultaneously, without enterI think what you'll notice is that they all (perhaps with the exception of Bellard's QuickJS) tend to omit this or that, extend this or that, not provide this or that capability, and might become resistant to developer feedback from the field sharing feedback the maintainers havn't thought of, or just flatly refuse to implement of see as "on-topic".

As to ECMAscript, and the minutae of what the technical writing means, and doesn't mean, some folks in the filed might have a different perspective, and different ideas. Top-down organizations might think they know best. The developer could decide to do their own thing, that leads to more stuff happening in the field, when the whole thing could be dealt with on a broader scale within the given entity/organization/specification body, where all stakeholders input MUST be included in the design pattern. As options.

I wouldn't mind ECMA-262 specifying reading standard input and standard output, as each engine and runtime implements that basic functionality differently. It's 2024, not 1995. Folks are running JavaScript - and now TypeScript - in all sorts of environments for multiple use cases. You would think STDIO would be interoperable and compatible. It ain't. So, there's diversity not by plan, rather by isolation, opinion, "Popularity".

You folks have a great day. Good luck!