ES modules not supported

[LinusU]

I'm getting the following error as soon as the compiled app boots:

node:internal/modules/cjs/loader:930
  throw err;
  ^

Error: Cannot find module '/snapshot/dhjaks/index.js'
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:927:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1776:46)
    at Function.Module._load (node:internal/modules/cjs/loader:772:27)
    at Function.runMain (pkg/prelude/bootstrap.js:1804:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Here is a minimal reproducible example:

package.json

{ "type": "module" }

index.js

import os from 'os'

console.log(os.arch())

Build command:

pkg index.js

[Gold-Samiar]

You didn't post your complete package.json file. So i think you didn't add your sub folders as scripts. For example if you want to add dhjaks folder as script folder then you need to add this in package file like. { "name": "mdm5", "version": "1.0.1", "description": "MDM", "main": "start.js", "bin": "start.js", "scripts": { "start": "node ." }, "pkg": { "scripts": [ "dhjaks/*.js" ], "assets": [], "targets": [ "node12", "linux-x64", "macos-x64", "win-x64" ] }, "author": "demo", "license": "ISC", "dependencies": { } } assume dhjaks is subfolder under pacakge file parent folder. use build command pkg ./package.json

[LinusU]

@sartaj-singh I actually did post my complete package.json file ☺️

The dhjaks folder is the folder of my entire package. My entire package only has two files: package.json & index.js. The goal is for it to just print out one line and then exit.

I did it like this to make a minimal test case that shows the problem.

---

I now tried to use pkg package.json instead:

$ mkdir foobar
$ cd foobar
$ echo '{ "name": "test", "bin": "index.js", "type": "module" }' > package.json
$ echo 'import os from "os"' > index.js
$ echo 'console.log(os.arch())' >> index.js
$ npx pkg package.json
> pkg@5.3.1
> Warning Failed to make bytecode node16-arm64 for file /snapshot/foobar/index.js

$ ./test
node:internal/modules/cjs/loader:930
  throw err;
  ^

Error: Cannot find module '/snapshot/foobar/index.js'
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:927:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1776:46)
    at Function.Module._load (node:internal/modules/cjs/loader:772:27)
    at Function.runMain (pkg/prelude/bootstrap.js:1804:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

[Gold-Samiar]

Check this line:- Warning Failed to make bytecode node16-arm64 for file /snapshot/foobar/index.js
don't use npm or npx, pkg can run independently. Try to set target in pacakge file like:- "targets": [ "node12", "linux-x64", "macos-x64", "win-x64" ]

[LinusU]

don't use npm or npx, pkg can run independently.

npx is just a way to install & run the package without clobbering your global installs. That is not what's causing problems here since I have tried it with a locally installed version of pkg as well.

Try to set target in pacakge file like

Setting the targets doesn't change anything, I've tried different targets and even running on different platforms...

It does however run if I don't use "type": "module", and use require instead of import, so this issue is clearly related to that.

[Gold-Samiar]

I always create const with require. import statement may be not supported by pkg. I think no need type=module if you compile stand alone executable.

[LinusU]

import statement may be not supported by pkg

If it isn't, then this is a feature request

I think no need type=module if you compile stand alone executable.

I need it because I need to import packages which are ESM-only

[webhype]

Hi, could someone clarify clearly on the home page (README.md) whether or not pkg supports ESMs (ES modules) at all? I know it's a free open-source labor-of-love project so I am not demanding anything. It is what it is and it is appreciated as-is. Just would like a clear positioning so we don't need to waste our time trying to package "type": "module" projects, if that's not supported at all. ESMs are not exactly a new invention so a one-liner positioning in the docs would be helpful. If ESM packaging is hopeless with pkg, does anyone know of a workaround (other than rewriting all your code back into CommonJS)? Cheers!

[CleyFaye]

There is the option of using a barebone webpack config to create a single JS file containing all dependencies and not having any external import. Something like this:

const config = {
  mode: "production",
  entry: "./src/main.ts",
  target: "node",
  output: {
    path: resolve(__dirname, "build", "lib"),
    chunkFormat: "commonjs",
  },
};

The output is then usable with pkg.

It should also be possible to update pkg to support ESM; last time I checked I saw two main issues, the babel configuration used (which can be either completely dropped or updated to support module input with a single change), and bytecode generation that failed. Since I already knew of the webpack option I gave up, but fixing bytecode generation with ESM should be doable since node now have full support for it.

[robertsLando]

For anyone interested I suggest you to firstly use ncc to compile your modules and then use pkg to compile them into executable. There is already an open feature request to include ncc in pkg, maybe with an option

[CleyFaye]

That was what we were doing until a recent update of ncc added compatibility with module-based source. It now produce files that pkg can't use; I could restore the build setup to get the actual error message if needed, but it was something along the line of not handling import statement that were indeed found in the output of ncc.

[robertsLando]

added compatibility with module-based source

Cannot this be disabled with an option?

[CleyFaye]

Not with an option, sadly. But in the end, ncc basically wraps webpack, hence our solution above. I'm not sure which of the two tools should change, but as it is some features of pkg are simply not used (bundling packages, detecting __dirname, etc.). Still the main feature works perfectly, so it's not so bad.

[robertsLando]

By double checking the code seems import statements should be supported: https://github.com/vercel/pkg/blob/59125d1820cdb380f3f592604bcfd7c017495e77/lib/detector.ts#L258

Maybe something isn't working as expected

[robertsLando]

I tried to look into this but haven't find the root cause, the build process seems to work as the import statement is recognized correctly but then the produced binary isn't working 🤷🏼‍♂️

[CleyFaye]

The exact issue, on a very minimalist project:

• have "type":"module" and "bin":"main.js" in package.json
• have import fs from "fs"; in main.js
• run pkg .

It will output this:

> pkg@5.3.2
> Targets not specified. Assuming:
  node16-linux-x64, node16-macos-x64, node16-win-x64
> Warning Failed to make bytecode node16-x64 for file /snapshot/t/main.js
> Warning Failed to make bytecode node16-x64 for file /snapshot/t/main.js
> Warning Failed to make bytecode node16-x64 for file C:\snapshot\t\main.js

And the binaries are unusable:

node:internal/validators:119                                                                                                                                               
    throw new ERR_INVALID_ARG_TYPE(name, 'string', value);                                                                                                                 
    ^                                                                                                                                                                      

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received null                                                                                
    at new NodeError (node:internal/errors:371:5)                                                                                                                          
    at validateString (node:internal/validators:119:11)                                                                                                                    
    at Object.basename (node:path:1309:5)                                                                                                                                  
    at Error.<anonymous> (node:internal/errors:1462:55)                                                                                                                    
    at getMessage (node:internal/errors:421:12)                                                                                                                            
    at new NodeError (node:internal/errors:348:21)                                                                                                                         
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1128:19)                                                                                            
    at Module.load (node:internal/modules/cjs/loader:981:32)                                                                                                               
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)                                                                                                     
    at Function.runMain (pkg/prelude/bootstrap.js:1804:12) {                                                                                                               
  code: 'ERR_INVALID_ARG_TYPE'                                                                                                                                             
}

Removing "type":"module" and altering the file to use require() produce a working build (but is not acceptable on a large codebase). Removing "type":"module" while keeping import statement won't work: error while generating bytecode, and the binary output:

(node:288927) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `t-linux --trace-warnings ...` to show where the warning was created)
/snapshot/t/main.js:1
import fs from "fs";
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at Object.compileFunction (node:vm:354:18)
    at wrapSafe (node:internal/modules/cjs/loader:1031:15)
    at Module._compile (node:internal/modules/cjs/loader:1065:27)
    at Module._compile (pkg/prelude/bootstrap.js:1758:32)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.runMain (pkg/prelude/bootstrap.js:1804:12)
    at node:internal/main/run_main_module:17:47

as expected.

And since ncc was brought up, using ncc on this minimal example and then using its output with pkg yields:

> pkg@5.3.2
> Targets not specified. Assuming:
  node16-linux-x64, node16-macos-x64, node16-win-x64
> Error! import.meta may appear only with 'sourceType: "module"' (5:95)
  /home/cleyfaye/t/dist/index.js

which I traced back to the babel config, and prevent the binary from being build. Quick-fixing this config issue brings us back to the issues described above without using ncc.

[jhmaster2000]

That was what we were doing until a recent update of ncc added compatibility with module-based source. It now produce files that pkg can't use; I could restore the build setup to get the actual error message if needed, but it was something along the line of not handling import statement that were indeed found in the output of ncc.

Would you happen to know what specific version of ncc made this change? I am also facing the issue addressed in this issue and am wondering if we could not just downgrade to an ncc version prior to that change and use that?

[CleyFaye]

The change was introduced in with ncc@0.29.0. Since we stopped using it I can't tell if something changed in later releases though.

[ForbiddenEra]

I haven't dug too deep.. but it looks like pkg wraps whatever program/package is compiled?

@ https://github.com/vercel/pkg/blob/main/prelude/bootstrap.js#L1845

  Module.runMain = function runMain() {
    Module._load(ENTRYPOINT, null, true);
    process._tickCallback();
  };

A minimal test using _load shows:

#~/test$ node testloader.js
node:internal/modules/cjs/loader:1146
      throw err;
      ^

Error [ERR_REQUIRE_ESM]: require() of ES Module ~/test/src/test.js not supported.
Instead change the require of test.js in null to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (~/test/testloader.js:2:8) {
  code: 'ERR_REQUIRE_ESM'
}

Node.js v17.2.0
#~/test$ cat testloader.js
const Module = require('module')
Module._load('./src/test.js',null,true)

So, would we not need to detect here if it's a "type":"module" in package.json or a *.mjs and then import it instead?

Is there a reason it's wrapped this way instead of execing node on the main script? Or is it actually even wrapped like that in the final package? Like I said, I haven't picked too deep on this issue yet but I'd like to help solve it if I can.

[robertsLando]

@ForbiddenEra if you check linked pr #1323 you will find the reason while esm are not supported yet

[ForbiddenEra]

@ForbiddenEra if you check linked pr #1323 you will find the reason while esm are not supported yet

I did read all of that, I guess I just (and am still not entirely) don't have full grasp on the process pkg is using. I do plan on possibly pulling the source and digging deeper though.

Now, even if the package was resolved correctly, would we not need a separate runMain for es modules..?

Or, is it the resolver actually generating said runMain function..? or..?

It would be nice if there was a list somewhere of the steps pkg takes exactly, ie:

1. parse files
2. lint/compile/byte code
3. compress

and with which libs/modules any step would involve. pkg seems to work quite differently than I might have guessed, ie, I would've thought that simply it created a self-extracting archive of a node setup and then simply run that node on the script, but there's obviously much more going on here.

[robertsLando]

I would've thought that simply it created a self-extracting archive of a node setup and then simply run that node on the script, but there's obviously much more going on here.

It's much more complicated then that, caxa does that (but doesn't provide source code protection). For more informations about how it work I have write a developer guide here: https://github.com/vercel/pkg/wiki/Developers

Based on what I have understand the only problem is we are using resolve package to resolve modules but it doesn't support es modules, we should use enhanced-resolve instead. Once that is done es modules should work

[ForbiddenEra]

I would've thought that simply it created a self-extracting archive of a node setup and then simply run that node on the script, but there's obviously much more going on here.

For more informations about how it work I have write a developer guide here: https://github.com/vercel/pkg/wiki/Developers

Awesome, I must've missed the link to that, I'll check it out.

[Nisthar]

guys does pkg works if you use dynamic imports?

[jesec]

This is not necessarily a bug.

However, this would be our highest priority feature request.

[lukaskellerstein]

Any progress here?

[robertsLando]

Follow updates on #1323. I know @jesec will try to implement it once he has some free time

[jesec]

[robertsLando]

Well done @jesec 🚀

[robertsLando]

@jesec Did you opened a PR with that?

[jesec]

No. Still long way to go at this point.

We might have to use vm.Module which is in experimental status as of Node 18.2. I am generally against relying on experimental API in this project. Additionally, bytecode generation and walking of async import is still unresolved at the moment.

[ForbiddenEra]

keep up the good work!

[student020341]

There is the option of using a barebone webpack config to create a single JS file containing all dependencies and not having any external import. Something like this:

const config = {
  mode: "production",
  entry: "./src/main.ts",
  target: "node",
  output: {
    path: resolve(__dirname, "build", "lib"),
    chunkFormat: "commonjs",
  },
};

Thanks! Hello from the future - I am trying to package a sveltekit application (front end + SSR) in an executable to hand off to a friend so they won't have to install nodejs or anything like that. I encountered an error trying to pkg the output of webpack when it produced multiple files because it creates dynamic imports. So in case anyone is lead to this issue from google, this workaround does work still, but with this change if your webpack is producing multiple files:

const webpack = require("webpack");

const config = {
  mode: "production",
  entry: "./src/main.ts",
  target: "node",
  output: {
    path: resolve(__dirname, "build", "lib"),
    chunkFormat: "commonjs",
  },
  plugins: [
    new webpack.optimize.LimitChunkCountPlugin({
      maxChunks: 1
    })
  ]
};

There are probably consequences for doing this, but it works for now!

[Inrixia]

@jesec Any update on the status of this?

[SoCuul]

Is there any update on this?

[Inrixia]

Is there any update on this?

No ESM support yet but I did manage to get my ESM project to compile by transpiling it to CJS first using esbuild. It works flawlessly though I think if you have any top level async it might break.

Here is the command I use:

esbuild ./src/float.ts --bundle --platform=node --outfile=dist/float.cjs && pkg ./dist/float.cjs --out-path=./build -t latest-linux,latest-mac,latest-win --compress GZip

[zekefeu]

Is there any update on this?

No ESM support yet but I did manage to get my ESM project to compile by transpiling it to CJS first using esbuild. It works flawlessly though I think if you have any top level async it might break.

Here is the command I use:

esbuild ./src/float.ts --bundle --platform=node --outfile=dist/float.cjs && pkg ./dist/float.cjs --out-path=./build -t latest-linux,latest-mac,latest-win --compress GZip

Works great on my end so far, even with javascript-obfuscator

[Nantris]

That's a great workaround @Inrixia. Unfortunately it doesn't work with top level await. If anyone knows a solution, I'd love so much to avoid having to use Webpack or Rollup to bundle a couple simple scripts.

Related issue: https://github.com/evanw/esbuild/issues/253

[MidKnightXI]

Another workaround that I'm using here is using rollup npm package like that:

rollup -c && pkg -o program-win -t node16-win-x64 bundle.js

EDIT: I switched to Rust so I don't use Rollup anymore went from 15mb package to 1.5mb. Choose the right tool for the right job

EDIT2: if you still want to see how I used Rollup: https://github.com/MidKnightXI/opgg-ads-remover/commit/70e76ef5b37da6e21d41c28324fd0be39ce99808

[Nantris]

I didn't think Rollup could handle top level await. I finally just removed top level await as it was too painful to build.

[rigwild]

No ESM support yet but I did manage to get my ESM project to compile by transpiling it to CJS first using esbuild. It works flawlessly though I think if you have any top level async it might break.

Here is the command I use:

esbuild ./src/float.ts --bundle --platform=node --outfile=dist/float.cjs && pkg .

I had an issue with this when using meow which asks for import.meta. you can use the Define esbuild API to make it work.

esbuild bin/cli.js --bundle --platform=node --outfile=dist/cli.cjs --define:import.meta.url=__dirname

#!/usr/bin/env node
// @ts-check

import meow from 'meow'

let url = import.meta.url
// Allow rewriting `import.meta.url` to `__dirname` when bundling with esbuild
if (!url.startsWith('file://')) url = new URL(`file://${import.meta.url}`).toString()

const cli = meow(`
  Usage
    $ your-cli
`,
  {
    // importMeta: import.meta,
    importMeta: { url },
  }
)

Then I could build without any issue.

[cspotcode]

We might have to use vm.Module which is in experimental status as of Node 18.2. I am generally against relying on experimental API in this project. Additionally, bytecode generation and walking of async import is still unresolved at the moment.

Can --loader help here, once it's stable?

[ctjlewis]

Yeah, transpiling to CJS is not stable because of TLA, actual ES modules will contain top-level await.

That's a great workaround @Inrixia. Unfortunately it doesn't work with top level await. If anyone knows a solution, I'd love so much to avoid having to use Webpack or Rollup to bundle a couple simple scripts.

There is no simple solution except adding bona fide ESM support. Thank you to package maintainers for their very important hard work on this.

[si458]

for anybody intrested this is how i got ESM working

1. use node 18 and set file extensions to mjs (index.mjs)
2. build/convert ESM to CJS using esbuild esbuild index.mjs --bundle --platform=node --target=node18 --outfile=out.js
3. set bin in package.json to out.js
4. use pkg to bundle new CJS file pkg . --no-bytecode --public-packages '*' --public
5. (optional) create icon.ico and must be less than 135kb
6. (optional) use resedit-cli to add custom windows data, icon, certifcate etc resedit --in myapp.exe --out out.exe --company-name "My Company" --file-description "My App Does Stuff" --file-version 1.0.0.0 --icon 1,icon.ico --no-grow --pfx certificate.pfx --password mysecretpassword --product-name "MyApp" --product-version 1.0.0.0 --sign --timestamp "http://timestamp.sectigo.com"

[CleyFaye]

That's basically the same approach as many above, while removing bytecode generation and basically removing all dynamic package loading since it's all bundled in one file. At this point it feels like that bypass most of the features of pkg, aside from the node bootstrap part.

[cedx]

[Off topic] Interesting to see that Node 20's single executable applications suffer from the same limitation.

The single executable application feature currently only supports running a single embedded script using the CommonJS module system.

[ctjlewis]

[Off topic] Interesting to see that Node 20's single executable applications suffer from the same limitation.

The single executable application feature currently only supports running a single embedded script using the CommonJS module system.

I also noticed this. I imagine they'll support ESM within 6 months, it's still experimental.

For now, it should be possible to get a bundle that works with SEAs using ESBuild --bundle option with CJS target and Node module resolution, same as here with pkg. You will just need to refactor any top-level await statements in your program.

[BLACK4585]

Hey, I'm new to PKG and have the problem with ES modules right now. I wrote my code in ts. So I assume I have to compile it to JS and then wrap it using pkg? I'm using to-level await in my code. The big problem which I have right now is the conversion or what I have to do exactly. I think I have to compile it to JS to sth like es3, so I don't have those async/await functions in my code anymore. But the tsc gives me a few errors, one of them is I can only use top-level await in newer versions. Is there any discord server or so to chat, is a bit easier than in here ig.

Thanks in advance!

[MidKnightXI]

Hey, I'm new to PKG and have the problem with ES modules right now. I wrote my code in ts. So I assume I have to compile it to JS and then wrap it using pkg? I'm using to-level await in my code. The big problem which I have right now is the conversion or what I have to do exactly. I think I have to compile it to JS to sth like es3, so I don't have those async/await functions in my code anymore. But the tsc gives me a few errors, one of them is I can only use top-level await in newer versions. Is there any discord server or so to chat, is a bit easier than in here ig.

Thanks in advance!

Just use tsc then rollup to convert it to cjs, you should then be able to use pkg.

[BLACK4585]

Hey, I'm new to PKG and have the problem with ES modules right now. I wrote my code in ts. So I assume I have to compile it to JS and then wrap it using pkg? I'm using to-level await in my code. The big problem which I have right now is the conversion or what I have to do exactly. I think I have to compile it to JS to sth like es3, so I don't have those async/await functions in my code anymore. But the tsc gives me a few errors, one of them is I can only use top-level await in newer versions. Is there any discord server or so to chat, is a bit easier than in here ig. Thanks in advance!

Just use tsc then rollup to convert it to cjs, you should then be able to use pkg.

So the following steps:

• tsc index.ts without any TS configuration
• rollup index.js --file bundle.js --format cjs to bundle everything together
• pkg bundle.js --targets node18-win-x64,node18-linux-arm64 to create the executable?

[BLACK4585]

In short: Doesn't work xD General improvement: There should be a public chatroom like discord, issues are not made for helping people like me doing things pkg doesn't support out of the box. When I bundle things together, it only bundles my node_modules with it with a few plugins, then pkg refuses to generate bytecode and so on. So, my "simple" question: What do I have to do to make pkg work? For me, it looks like I'm very restricted when it comes to writing code that works with pkg because I can't use import x from x or so. When I then try to bundle it or first just compile it to JS which pkg understands, errors about top level await arise…