Resolving tsconfig paths breaks inside node_modules

IlyaSemenov commented 1 year ago

Bug description

I put my project inside node_modules directory (not even real node_modules alongside some package.json, just a new directory named that).

After doing so, resolving tsconfig paths stopped working.

I expect tsx to be agnostic to where the project is located (as long as it has its own package.json and tsconfig.json), same as esbuild is.

Reproduction

I repeated this on Stackblitz: https://stackblitz.com/edit/node-qjlxvn

This does not completely resemble the description above (it uses a 'real' node_modules directory) but the very same code can be used to repeat the problem on a completely empty node_modules.

Run:

~/projects/node-qjlxvn
❯ cd test

~/projects/node-qjlxvn/test
❯ tsx index.ts
value

~/projects/node-qjlxvn/test
❯ cd ..

~/projects/node-qjlxvn
❯ mv test node_modules/

~/projects/node-qjlxvn
❯ cd node_modules/test

~/projects/node-qjlxvn/node_modules/test
❯ tsx index.ts
Error: Cannot find module '@/value'
Require stack:
- /home/projects/node-qjlxvn/node_modules/test/index.ts
    at Module._resolveFilename (https://nodeqjlxvn-s25i.w-credentialless.staticblitz.com/blitz.b6c96f782a49b3e017dca41830943768f8acbe40.js:6:217308)
    at d.default._resolveFilename (file:///home/projects/node-qjlxvn/node_modules/.pnpm/@esbuild-kit+cjs-loader@2.4.1/node_modules/@esbuild-kit/cjs-loader/dist/index.js:1:1554)
    at Module._load (https://nodeqjlxvn-s25i.w-credentialless.staticblitz.com/blitz.b6c96f782a49b3e017dca41830943768f8acbe40.js:6:214847)
    at Module.require (https://nodeqjlxvn-s25i.w-credentialless.staticblitz.com/blitz.b6c96f782a49b3e017dca41830943768f8acbe40.js:6:218087)
    at i (https://nodeqjlxvn-s25i.w-credentialless.staticblitz.com/blitz.b6c96f782a49b3e017dca41830943768f8acbe40.js:6:415284)
    at _0xc8b09a (https://nodeqjlxvn-s25i.w-credentialless.staticblitz.com/blitz.b6c96f782a49b3e017dca41830943768f8acbe40.js:15:142803)
    at eval (file:///home/projects/node-qjlxvn/node_modules/test/index.ts:2:774)
    at Object.eval (file:///home/projects/node-qjlxvn/node_modules/test/index.ts:3:3)
    at Object.function (https://nodeqjlxvn-s25i.w-credentialless.staticblitz.com/blitz.b6c96f782a49b3e017dca41830943768f8acbe40.js:15:143540)
    at Module._compile (https://nodeqjlxvn-s25i.w-credentialless.staticblitz.com/blitz.b6c96f782a49b3e017dca41830943768f8acbe40.js:6:219079) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/home/projects/node-qjlxvn/node_modules/test/index.ts' ]
}

~/projects/node-qjlxvn/node_modules/test
❯ esbuild --bundle index.ts
(() => {
  // src/value.ts
  var value_default = "value";

  // index.ts
  console.log(value_default);
})();

Environment

System:
    OS: Linux 5.0 undefined
    CPU: (8) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Memory: 0 Bytes / 0 Bytes
    Shell: 1.0 - /bin/jsh
  Binaries:
    Node: 16.14.2 - /usr/local/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 7.17.0 - /usr/local/bin/npm
  npmPackages:
    tsx: ^3.12.1 => 3.12.1

Can you contribute a fix?

[ ] I’m interested in opening a pull request for this issue.

IlyaSemenov commented 1 year ago

Of course this is arguably an edge case scenario, what I am really struggling with is that I can't add a new test dependency under tests/fixtures/tsconfig/node_modules/my-new-test-dependency with its tsconfig.json (for #96) - it ignores its tsconfig.json altogether even if I run it directly (not as a dependency). I then narrowed the problem to having node_modules in path.

zocky commented 1 year ago

I have this strange behavior:

project/node_modules/dependency breaks tsx in dependency
project/node_modules/<symlink to dependency> doesn't break it, but, if the dependency is located within a node_modules directory, then it breaks again.

zocky commented 1 year ago

This is caused explicitly by these two lines in cjs-loader and esm-loader.

If you disable those lines, tsx works as expected, at least for my use case.

privatenumber commented 1 year ago

Doesn't sound related to the issue being reported here. Please file a separate issue.

zocky commented 1 year ago

No, it's the exact same issue. Any file that has node_modules in its real path will ignore tsconfig, because esm-loader and cjs-loader will refuse to apply tsconfig.json to it.

privatenumber commented 1 year ago

Elaborating on how your code is breaking (what code? what error?) would add more value to this thread. Right now, it's hard for readers to understand the problem you're experiencing.

Does removing those two lines fix it for the reproduction provided?

If you feel strongly that it's the same issue, that's fine. But if this gets fixed and doesn't resolve your issue, you'll have to open a new issue. Always better to file earlier so it's on my radar & roadmap.

zocky commented 1 year ago

The difference in behavior when the parent path contains the string /node_modules/ definitely seems to be caused by those two lines, so solving this will solve my issue.

First, my usecase - I'm writing a website generator which is supposed to be installed with npm and run with npx, and to import files from the main package directory.

This now breaks my paths configuration if my generator package is properly installed (i.e. not symlinked), because the files that the start script loads are now within a node_modules directory, so tsx refuses to use my tsconfig.json to resolve paths, even if I run the script or tsx itself manually from that directory.

My current development work around is mv node_modules mode_nodules; ln -s mode_nodules node_modules. This "works", i.e. the start script runs tsx which correctly uses paths from the specified tsconfig to resolve modules.

The drawback is that all modules from all packages are now loaded using the same logic specified in the originally used tsconfig. This hasn't broken anything in my project so far, but it has made the load time a tiny bit longer. (Interestingly, it also gave me a warning which found a bug in jsonpath. I must report it after I get some sleep.)

I can see two ways of fixing this:

Always use the "local" tsconfig (i.e. one that's not above the closest package.json).
Use the originally specified tsconfig.json for files within the package where tsx was started (i.e. walk up from cwd to the nearest package.json, any file that's within that directory uses tsconfig paths for resolving modules), regardless of whether they are within a node_modules directory.

IlyaSemenov commented 1 year ago

Any reason for publishing a non-compiled version to npm that needs a real-time bundler to run? That's very untypical. I believe in your case it makes sense to use e.g. tsup and publish cjs and mjs versions which will start natively.

zocky commented 1 year ago

The website generator imports jsx files which need to be transpiled before import, so using tsx at runtime is already a part of the project.

I realize that there must be a way to accomplish that without loading the whole thing through tsx every time it's started. But using tsx as a drop-in replacement for node would make it much more comfortable, and the expense is really just compiling several additional files at startup.

Edit: It's actually a web server, not really a website generator. It imports some jsx files at startup and can import new files during runtime, so compiling the whole site at startup is not an option.

shigma commented 9 months ago

I'm encountered the same issue. In the above screenshot, the two entry files have exactly the same content, but only the one outside node_modules can be loaded with paths support.

https://github.com/privatenumber/tsx/blob/985bbb8cff1f750ad02e299874e542b6f63495ef/src/esm/loaders.ts#L155

I haven't studied the source code carefully, but I want to ask what is this line for? Will modifying the logic here help solve this issue? (Actually I can't even find the relavant code in dist, so I have no idea how to test on it.)

enisdenjo commented 4 hours ago

Encountered the same issue now. Removing this line helped solve it:

https://github.com/privatenumber/tsx/blob/985bbb8cff1f750ad02e299874e542b6f63495ef/src/esm/loaders.ts#L155

I'd either remove it completely - I don't see it being a big performance bottleneck as the resolution is executed only once per import; or at least make it a flag when running tsx, something like --allow-tsconfig-paths-in-node_modules.

@privatenumber your input would be grately appreciated here. If you're still struggling on understanding how this exactly happens, I'd be happy to create you a repro.

privatenumber / tsx