ERR_UNKNOWN_FILE_EXTENSION when using typescript

vedantroy commented 2 years ago

I am getting the following error when using typescript:

tests/PDFViewer.ts 2> TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /home/vedantroy/Desktop/pdf-testing/pdf-svelte/tests/PDFViewer.ts

Here's my package.json (ignore the ava stuff):

{
  "name": "pdf-svelte",
  "version": "0.0.1",
  "scripts": {
    "dev": "svelte-kit dev",
    "build": "svelte-kit build",
    "package": "svelte-kit package",
    "preview": "svelte-kit preview",
    "check": "svelte-check --tsconfig ./tsconfig.json",
    "check:watch": "svelte-check --tsconfig ./tsconfig.json --watch",
    "format": "prettier --write --plugin-search-dir=. ./**/*.{svelte,ts,json}",
    "test": "tap --ts"
  },
  "devDependencies": {
    "@ava/typescript": "^3.0.1",
    "@babel/preset-typescript": "^7.16.7",
    "@sveltejs/adapter-auto": "next",
    "@sveltejs/kit": "next",
    "@types/tap": "^15.0.5",
    "ava": "^4.0.1",
    "prettier": "^2.5.1",
    "prettier-plugin-svelte": "^2.6.0",
    "svelte": "^3.44.0",
    "svelte-check": "^2.2.6",
    "svelte-preprocess": "^4.10.1",
    "tap": "^15.1.6",
    "ts-node": "^10.5.0",
    "tslib": "^2.3.1",
    "typescript": "~4.5.4"
  },
  "type": "module",
  "dependencies": {
    "pdfjs-dist": "^2.12.313"
  },
  "ava": {
    "typescript": {
      "rewritePaths": {
        "src/": "test-artifacts/"
      },
      "compile": false
    }
  }
}

Is this because I have a custom tsconfig.json due to using Svelte?

{
  "compilerOptions": {
    "moduleResolution": "node",
    "module": "es2020",
    "lib": ["es2020", "DOM"],
    "target": "es2020",
    /**
            svelte-preprocess cannot figure out whether you have a value or a type, so tell TypeScript
            to enforce using \`import type\` instead of \`import\` for Types.
        */
    "importsNotUsedAsValues": "error",
    /**
            TypeScript doesn't know about import usages in the template because it only sees the
            script of a Svelte file. Therefore preserve all value imports. Requires TS 4.5 or higher.
        */
    "preserveValueImports": true,
    "isolatedModules": true,
    "resolveJsonModule": true,
    /**
            To have warnings/errors of the Svelte compiler at the correct position,
            enable source maps by default.
        */
    "sourceMap": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "baseUrl": ".",
    "allowJs": true,
    "checkJs": true,
    "paths": {
      "$lib": ["src/lib"],
      "$lib/*": ["src/lib/*"]
    }
  },
  "include": ["src/**/*.d.ts", "src/**/*.js", "src/**/*.ts", "src/**/*.svelte"]
}

isaacs commented 2 years ago

It's because you have "type": "module" in your package.json, I believe. Does it work if you remove that?

IIUC, node will only load files ending in .js, .cjs and .mjs when in a "type": "module" project.

vedantroy commented 2 years ago

It's because you have "type": "module" in your package.json, I believe. Does it work if you remove that?

IIUC, node will only load files ending in .js, .cjs and .mjs when in a "type": "module" project.

I generated this project using SvelteKit, so I am hesitant to remove the type: module since that was created by Svelte.

However, I'll check if removing type: module works.

vedantroy commented 2 years ago

It's because you have "type": "module" in your package.json, I believe. Does it work if you remove that?

IIUC, node will only load files ending in .js, .cjs and .mjs when in a "type": "module" project.

Update: I can't remove the "type": "module" because that is used by Svelte. Is there a work-around?

vedantroy commented 2 years ago

I wanted to check in on whether there was a way to use node-tap with "type": "module". Currently I'm using the following hack which compiles my entire project to JS using esbuild and then runs the rest runner on the compiled output (the script works with ava, but it could also be easily adapted for node-tap):

#!/bin/bash

# Ended up taking the "brute-force" approach:
# - Compile the entire project to JS
# - Run AVA (with no Typescript configuration) on the compiled JS project

TEST_DIR=.test-artifacts
rm -rf $TEST_DIR
echo "Finished: Clean test artifacts"

src_files='./src/**/*.ts'
test_files='./tests/*.ts'
npx esbuild --outdir=$TEST_DIR --bundle --platform=node --external:ava $test_files $src_files
echo "Finished: Compile src/test files to JS"

# I do not know why this is necessary ...
files="./${TEST_DIR}/tests/*.js"
for i in $files; 
    # TODO: Probably a cleaner way to do this
    do mv $i $(dirname $i)/$(basename -s js $i)cjs; 
done
echo "Finished: Change test files to .cjs extension"

test_opt=$1
shift

# Bash is hard ...
if [ "$test_opt" = "debug" ]; then
  eval "npx ava debug ${TEST_DIR}/**/*.cjs ${@}"
elif [ "$test_opt" = "normal" ]; then
  eval "npx ava ${TEST_DIR}/**/*.cjs ${@}"
else
  echo "Error: Unknown test mode $1"
  exit 1
fi

isaacs commented 2 years ago

That's certainly one way to do it 😅

I'd put a set -e in your bash script, it'll make your life a lot more pleasant. (set -e tells bash to exit the script on errors, rather than continuing on in an undefined state.)

vedantroy commented 2 years ago

That's certainly one way to do it 😅

I'd put a set -e in your bash script, it'll make your life a lot more pleasant. (set -e tells bash to exit the script on errors, rather than continuing on in an undefined state.)

Yeah the final version has set -euxo pipefail Is there maybe some hacky workaround that I can use to avoid a shell script? Or maybe a small patch to node-tap?

Fwiw; all svelte-kit projects have type: module, and while svelte isn't a super popular framework like react it might be worth supporting.

isaacs commented 2 years ago

Yeah, honestly I just haven't gotten around to combing through all the combinations, and ts+type:module seems to be a broken one. Not sure what the way forward is there (but I'm guessing there is one where it Just Works, maybe with some changes to tap). The files are being executed by tap with ts-node, right? If you write a super minimal test.ts that just imports tap and does t.pass(), does that run fine?

You might also find some more clues putting NODE_DEBUG=tap in the environment to see where exactly it's falling over.

If you do go the "compile and test as js" route, you could put the compilation in a pretest script in your package json, or in a node module set as tap's before config, just to save the hassle of running a bash script manually, at least.

isaacs commented 2 years ago

Ok, I think I figured out what needs to happen: https://github.com/tapjs/ts-esm-testing/tree/main (This page was very helpful!)

Don't use node -r ts-node/register ${file}. We must run it with the ts-node-esm binary. node ${tsNodeESM} ${file}

The project tsconfig.json should contain this:

{
  "compilerOptions": {
    "module": "ES2020",
    "moduleResolution": "node"
  }
}

Tests have to import files using import {myModule} from '../index.js' (even though the actual file is named ../index.ts), because node's ESM loader insists on full actual filename paths, without their extensions truncated.

Tap can't help you with adding the .js to your imports of .ts files, that's in your test code. Oh well.

But what we can do is:

Load *.@(ts|tsx) files as node ${tsNodeESM} ${file} instead of node -r ${tsNode} ${file}. (This seems to work fine with non-ESM typescript as well.)
Put TS_NODE_COMPILER_OPTIONS='{"module":"ES2020","moduleResolution":"node"}' in the environment. (Merged in with any existing TS_NODE_COMPILER_OPTIONS, of course.)

isaacs commented 2 years ago

Another wrinkle: if you do not have "type": "module" in package.json, then putting "module":"ES2020" in TS_NODE_COMPILER_OPTIONS makes it fail with:

(node:5001) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `node --trace-warnings ...` to show where the warning was created)
(node:5001) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
/Users/isaacs/dev/tapjs/tap/ts-esm/test/basic.ts:37
import { t, f, invertLogic, fixLogic, breakLogic, } from '../index';
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at Object.compileFunction (node:vm:352:18)
    at wrapSafe (node:internal/modules/cjs/loader:1026:15)
    at Module._compile (node:internal/modules/cjs/loader:1061:27)
    at Module.m._compile (/Users/isaacs/dev/tapjs/tap/ts-esm/node_modules/ts-node/src/index.ts:1455:23)
    at Module._extensions..js (node:internal/modules/cjs/loader:1151:10)
    at Object.require.extensions.<computed> [as .ts] (/Users/isaacs/dev/tapjs/tap/ts-esm/node_modules/ts-node/src/index.ts:1458:12)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:168:29)
    at ModuleJob.run (node:internal/modules/esm/module_job:197:25)

So anyone using "type": "module" in package.json will have to also add "module":"ES2020" to either the TS_NODE_COMPILER_OPTIONS env or their tsconfig.json project file.

cspotcode commented 2 years ago

Thought I'd mention a couple ts-node features that might be relevant here:

As an alternative to TS_NODE_COMPILER_OPTIONS, users can also add "ts-node": {"compilerOptions": {/* overrides */}} to their tsconfig file. These achieve the same effect: they are overrides that only apply in ts-node, they don't affect tsc.

We also have moduleTypes which lets you simultaneously override package.json "type" and tsconfig "module" for files matching declared globs. https://typestrong.org/ts-node/docs/module-type-overrides

[Using ts-node-esm] seems to work fine with non-ESM typescript as well.

You can safely depend on this behavior; we intend to always register all hooks, including the CJS ones, when using the ESM loader. We can't do the inverse -- automatically register the ESM loader from -r ts-node/register -- because it requires passing --loader to a new process.

The ts-node-esm binary handles --loader, lets you pass all ts-node CLI flags, and suppresses node's experimental loader warning. But you also have the option of passing --loader ts-node/esm if you want to avoid an extra process and don't need those conveniences.

isaacs commented 2 years ago

Thanks, @cspotcode!

Yes, if ts-node-esm runs Yet Another subprocess, then using the --loader approach would be much nicer, I'll do that instead.

As an alternative to TS_NODE_COMPILER_OPTIONS, users can also add "ts-node": {"compilerOptions": {/ overrides /}} to their tsconfig file.

I try to avoid editing files during the test run, because that has a high potential for user surprise if it doesn't get put back the way it was. But it's good to know that there's another tsconfig field I should be checking for user intent. If they put something there, tap should respect it (and then clobber it with just the stuff that we know the tests will need to run at all).

cspotcode commented 2 years ago

I am hopeful that node will eventually let us register loader hooks in-process so that the subprocess is not necessary. That's all pending how some loader composition design stuff shakes out.

I try to avoid editing files during the test run,

Definitely agreed. I am never sure if tool authors prefer to ask their users to configure ts-node on their own, or prefer to pass us appropriate overrides to ensure stuff "just works." Sorta similar to how, if a user had the wrong package.json "type", you might ask them to fix it on their own. ts-node's configuration can be thought of as falling into the same category: if it's wrong, the runtime fails.

isaacs commented 2 years ago

I have the basics of this working now, but coverage (and thus --watch and --changed) will remain 100% broken until I can switch from NYC to c8. Processinfo stuff happening over at https://github.com/tapjs/processinfo, which looks like a decent approach so far. The remaining sticky wickets:

t.mock() support for imports #808 #804
c8 produces a massive amount of coverage info (everything in node core and node_modules, don't see a way to exclude it). Could maybe save a lot of disk space by just pruning the data objects or something, if V8 doesn't give us a way to filter. More research needed.
Need to filter out that --experimental-loader warning, it's brutal when you have a lot of child processes running with it.

cspotcode commented 2 years ago

Need to filter out that --experimental-loader warning

That is one of the QoL things that ts-node --esm and ts-node-esm takes care of. Could look at our implementation.

isaacs commented 2 years ago

Relevant to these interests: https://www.npmjs.com/package/multiloader

er, this rather: https://github.com/cspotcode/multiloader

Ethan-Arrowood commented 2 years ago

I was able to get my TypeScript ESM project working by using tap --node-arg=--loader=ts-node/esm. But there is no coverage. You can see source here: https://github.com/jsperfdev/jsperf.dev/blob/56a141a3a6d3667e730bedca9fde552e5042c654/packages/benchmark/package.json

Any ideas? Honestly, I'm not tied to using ESM but thought I'd give it another shot to see how support has developed.

ekoeryanto commented 2 years ago

Halo, i found bob-ts package out there. with this, coverage is working.

"type": "module",
  "scripts": {
    "start": "node -r dotenv/config dist/server.js",
    "dev": "bob-ts-watch -i src -c \"npm start\"",
    "pretest": "bob-ts -f cjs -i test -d .tmp ",
    "test": "tap .tmp/**/*.test.cjs"
  },
  "tap": {
    "node-arg": [
      "--require=dotenv/config",
      "--no-warnings",
      "--experimental-loader=@istanbuljs/esm-loader-hook"
    ]
  },

isaacs commented 2 years ago

Another option is to use c8. Check out the scripts and tap settings here: https://github.com/isaacs/cli-env-config/blob/main/package.json#L37

This will be fixed as soon as I can find the time for it. It's annoying me to have to do all this junk. 😅

hayatae commented 2 years ago

Solution I'm currently using to do testing and code coverage:

"test": "NODE_OPTIONS=\"--loader ts-node/esm\" c8 -r html -r text tap --no-coverage",

Only issue I haven't figured out yet is that I can't get mocking to work when using typescript + module. Always results in MODULE_NOT_FOUND for some reason.

head-gardener commented 2 years ago

My final solution for ts, esm, coverage and mocking was

"tap": {
  "node-arg" : [
    "--loader=ts-node/esm",
    "--loader=esmock"
  ],
  "coverage": false,
  "ts": false
}

And

"test": "c8 tap test/"

Seems to work fine!

WhereJuly commented 1 year ago

@vedantroy, @isaacs

Loved tap a lot for its unique speed in TypeScript. Managed to make it working with TypeScript files with "type": "module" in the package.json (code coverage working as well, nyc in my case). Here is the extract of my configuration in this gist.

In short as compared with the above solutions @esbuild-kit/esm-loader made a trick for me.

node-arg:
  - --loader=@esbuild-kit/esm-loader

Hope it helps someone.

isaacs commented 1 year ago

If anyone is feeling brave, you can try out tap 18, which has full coverage support for all combinations of TypeScript, CommonJS, and ESM.

npm i tap@pre

The documentation is coming along, but it's not styled yet. https://node-tap.surge.sh https://node-tap.surge.sh/changelog/#18.0

In short as compared with the above solutions @esbuild-kit/esm-loader made a trick for me.

If you want to use esbuild-kit instead of ts-node, you can do this with tap 18:

tap plugin rm @tapjs/typescript
tap plugin add @tapjs/esbuild-kit

isaacs commented 1 year ago

Docs are up. Once there's a ts-node tap can use without pulling from a git dep (install is slowwwww until that happens, but it works fine) it'll get published to npm on the latest dist-tag. For now, npm i tap@pre to check it out.

tapjs / tapjs

ERR_UNKNOWN_FILE_EXTENSION when using typescript #807