Allow executing TS scripts when NPM package is internal & private, and are treated as "binaries" (with shebang, etc.)

castarco commented 1 month ago

Related issues

Problem description

I'm working in a monorepo environment, with many JS/TS packages, many of them private, and not published to any registry.
Some of these non-published private packages are automation scripts.
To improve cross-platform compatibility, I want to avoid relying on Bash or POSIX shell scripts. I have other reasons as well, such as maintainability, type checking, static analysis, etc.
These packages DO NOT export TypeScript modules to be consumed by JS/TS code elsewhere.
But they do contain TypeScript files with a shebang to ensure that they will run under the Deno runtime, something like
```
#!/usr/bin/env -S deno run --node-modules-dir --allow-env --allow-read --allow-sys --allow-run
```

I export the corresponding binary in the package.json file of said package:

{
"name": "@kindspells/internal-tools",
"private": true,
"type": "module",
"bin": { "biome-check-staged": "./src/biome-check-staged.ts" },
"dependencies": { "zx": "^8.1.4" }
}

I install this package as a dev dependency in other packages of this same monorepo, so I can use the automation scripts from there, without having to deal with relative paths and other nightmarish command line tricks.

When I run the exported binary with PNPM... Deno refuses to run and makes me feel sad:

error: TypeScript files are not supported in npm packages: file:///Users/me/mycode/node_modules/@kindspells/internal-tools/src/biome-check-staged.ts

As a side note, it doesn't matter if I try to trick the system by calling a .js file that in turn calls the .ts file.

Proposed solution

I am aware that there are other open issues that got closed because it is a very bad idea to allow people to pack TypeScript in NPM packages instead of distributing JS directly. I want to be clear that this is not what I'm asking for. I'll go into the details right now.

My first point would be to notice the fact that we are talking about a "binary", a script that it is intended to be executed "as is" (it even has its own shebang to tell the OS how to be executed). We are not talking about code distribution.

Now, my proposal is that, when all of the following conditions are met, Deno should allow the execution of TypeScript scripts even when they are exposed through NPM packages:

The package is in the same monorepo, configured as a "workspace"
The package is private ("private": true in package.json)
The TypeScript code is not exported as a module or anything like that, but as a binary with its own shebang.
- Exported through package.json's "bin" field.
- The file is marked as executable
- The file has its own shebang

We know that, if we decided to pack POSIX shell scripts in NPM packages, they would be executed without hesitation, even though they are not pure JavaScript. The same would happen if we decided to pack Python scripts. For consistency (and a better life for everyone) the same should apply to virtually any scripting language, including TypeScript.

I don't think that being in the same monorepo and being private are important requirements, but I introduced them here to minimise potential concerns, and because in any case... this is the most common use case.

Alternative Solution

I've been checking the source code to see where this error is raised, and although I still didn't invest enought time, I can see how much of the information I mentioned in the previous proposal could not be available at the time of deciding whether to accept or not that script execution.

A simpler alternative could be to pass a specific flag to deno run (in the shebang) that disables the check raising the error... but adding an extra check that verifies that there is a shebang in the executed flag, and raise an error if it's not there.

This second check once we are at runtime would serve to ensure that people do not abuse the config flag for "normal code", and it is used only in "binaries".

dsherret commented 1 month ago

It should work if you use --no-config and don't reference the package.json dependencies in the deno script.

dsherret commented 1 month ago

The package is private ("private": true in package.json)

Sorry, I don't think it's a good idea to support typescript in any npm package (private or not) until node stabilizes the behaviour they're going to chose.

You can get around this behaviour by using the flag I mentioned in my previous comment.

castarco commented 1 month ago

@dsherret It's not really about "supporting TypeScript", but about letting "self-contained" scripts(1) be executed as they were intended, and only when they are exposed through the bin entry in package.json (2), as if they were any other binary (the other proposed constrains were there only to minimise potential problems).

We can call them "directly", without having to write deno or node, because they are in charge of specifying the interpreter via their fixed in their shebang.
Never for modules loaded via import / required, and also never for "truly external" packages, only verifiably "local" ones.

P.S.: Regarding your proposed solution, using --no-config and no NPM dependencies, it works as you mentioned, but I still think it presents some problems:

it is much more difficult to track dependencies versions (yes, I can go with deps.ts file, or something along the lines, but it's still subpar)
I would have two on-disk caches, one for application code, and another one for TypeScript automation scripts.
It makes it much more difficult to share utility libraries between application code and my automation scripts.

castarco commented 1 month ago

In case anyone wants to do something similar to what I describe here, there are other possible hacks to bypass Deno restrictions (although it has some caveats that I will enumerate at the end of this post):

In ./src/myscript.ts:

#!/usr/bin/env -S deno run --node-modules-dir --unstable-byonm --allow-env --allow-read --allow-sys --allow-run

// --node-modules-dir: allows using modules declared in `package.json`'s `dependencies` field.
// --unstable-byonm: uses `node_modules` managed by other tools (npm, pnpm, ...)

import { $ } from 'zx'

// Do stuff

In the wrapper ./src/myscript.sh file:

#!/bin/sh

# Compatibility notes:
# - This script tries to stick to POSIX shell, not depending on
#   non-standard Bash features.
# - `realpath` is commonly available in Unix-like systems... but
#   it wasn't always like that. In Macos it's supported since v13.

# POSIX shell boilerplate to fail early
set -eu;

unset CDPATH; # For safety

# Obtain the directory where the script is placed
SCRIPT_DIR="$( cd -- "$( dirname -- "$0" )" &> /dev/null && pwd )";

# Resolve the absolute directory path, removing symlinks.
SCRIPT_DIR="$(realpath "${SCRIPT_DIR}")";

# Now we can call the TypeScript script without problems:
#   If the script is inside an NPM package that belongs to the same
#   monorepo and has not been installed from a remote registry, then
#   the absolute path won't contain the `node_modules` subpath, and
#   Deno will allow us to execute TS scripts.
"${SCRIPT_DIR}/myscript.ts" $@;

In package.json:

{
  "name": "@coderspirit/internal-tools",
  "private": true,
  "type": "module",
  "bin": {
    "myscript": "./src/myscript.sh"
  },
  "dependencies": {
    "zx": "^8.1.4"
  }
}

Caveats

ugly, complex, higher maintenance costs
works well in macos >= 13... but it fails in Github Actions.

denoland / deno