search

yao-pkg / pkg

Package your Node.js project into an executable
https://www.npmjs.com/package/@yao-pkg/pkg
MIT License
395 stars 19 forks source link

Cannot run a child process of a binary asset #92

Closed dkyeremeh closed 2 months ago

dkyeremeh commented 2 months ago

What version of pkg are you using?

5.11.5

What version of Node.js are you using?

20.17.0

What operating system are you using?

ubuntu

What CPU architecture are you using?

x64

What Node versions, OSs and CPU architectures are you building for?

node20-linux-x64

Describe the Bug

My application processes videos using the bento4 tools. I am using bento4-node npm package to install bento4 in my app.

I am able to process videos when using non-packaged app. But the moment I try to use the bento4 tools in the packaged app, I get an error spawn /snapshot/src/node_modules/bento4-node/src/platforms/linux-x64/bin/mp4fragment ENOENT.

It seems for some reason, child_process cannot execute binaries in the snapshot

My code looks something like this

const { promisify } = require("util");
const cp = require("child_process");
const execFile = promisify(cp.execFile);
const bento4 = require("bento4-node");

if(!bento4.isPlatformSupported) throw Error("Bento4 tools not supported on this platform");

async function fragmentFiles(src, dest) {
  await execFile(bento4.mp4fragment, [src dest]);
}

What I have done so far

  1. I have added node_modules/bento4-node to the assets in the pkg.assets in the package.json.
  2. I have confirmed that the binaries are added.
    1. I checked the output bundle size when the the binaries are specified in the assets vs when they are not.
    2. The code for bento4-node is designed not to return the binaries if they don't exist. T
  3. When I update the code to point to the bento4 binaries on my machine (not in the snapshot), it works perfectly.
  4. I've tried child_process.exec, execFile, and spawn, but none of them seem to works

Note

This issue does not happen for bento4 tools alone. Using packaged binaries for ffmpeg and ffprobe do not work either.

Expected Behavior

Binary file should be executed as expected.

To Reproduce

  1. Define a binary asset in pkg.assets section of package.json
  2. create some code that attempts to call the binary file
  3. You get an error that the binary file cannot be found
robertsLando commented 2 months ago

Please show me package.json content and also share pkg command output with --debug flag. Then run the result package with DEBUG_PKG=2 env var and share the run output as well

dkyeremeh commented 2 months ago

Thank you for the quick feedback.

I have run everything you requested. I did it on my dev machine which is running MacOS instead.

The error is /bin/sh: /snapshot/server-app/node_modules/@ffmpeg-installer/darwin-arm64/ffmpeg: No such file or directory\n

The output can be found at https://gist.github.com/dkyeremeh/d0c20722477569e87654be96f3947294

There is another error on the console. Maybe it might help (node:83712) [fs-extra-WARN0003] Warning: fs.realpath.native is not a function. Is fs being monkey-patched?

robertsLando commented 2 months ago

@dkyeremeh is your project a monorepo app? I have a feel the problem is the path where the binary is loaded, also in the assets try to use globs instead like /**/*

dkyeremeh commented 2 months ago

Are you saying I should attatch /**/* to the assets?

dkyeremeh commented 2 months ago

I have setup a very simple project (12 lines) to reproduce the error here https://github.com/dkyeremeh/pkg-debug. It's not not a monorepo, and I have added /**/* to the assets as you requested.

I am deleting the Gist to prevent any accidental sensitive data exposure.

dkyeremeh commented 2 months ago

This is the output when I run it with DEBUG_PKG=2

$ pkg index.js -o build/app
> pkg@5.15.0
> Targets not specified. Assuming:
  node20-macos-arm64

pkg/prelude/bootstrap.js:1872
      throw error;
      ^
Could not find ffmpeg executable, tried "/snapshot/pkg-debug/node_modules/@ffmpeg-installer/darwin-arm64/ffmpeg", "/snapshot/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/node_modules/@ffmpeg-installer/darwin-arm64/ffmpeg" and "/snapshot/pkg-debug/node_modules/@ffmpeg-installer/darwin-arm64/ffmpeg"
(Use `app --trace-uncaught ...` to show where the exception was thrown)

Node.js v20.17.0
error Command failed with exit code 1.

Here is the output of the pkg command with --debug

$ pkg index.js -o build/app --debug
> pkg@5.15.0
> Targets not specified. Assuming:
  node20-macos-arm64
> [debug] Bytecode of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/index.js is added to queue.
> [debug] Stat info of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/index.js is added to queue.
> [debug] Content of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/package.json is added to queue. It was required from /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/index.js
> [debug] Bytecode of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/index.js is added to queue. It was required from /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/index.js
> [debug] Directory /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug is added to queue.
> [debug] Stat info of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/package.json is added to queue.
> [debug] Stat info of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/index.js is added to queue.
> [debug] Path.resolve(__dirname.substr(0, __dirname.indexOf('node_modules')), 'node_modules', '@ffmpeg-installer', platform) is ambiguous
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/index.js
  It resolves relatively to 'process.cwd' by default, however
  you may want to use 'path.dirname(require.main.filename)'
> [debug] Path.resolve(__dirname, '..', platform) is ambiguous
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/index.js
  It resolves relatively to 'process.cwd' by default, however
  you may want to use 'path.dirname(require.main.filename)'
> [debug] Path.resolve(__dirname, 'node_modules', '@ffmpeg-installer', platform) is ambiguous
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/index.js
  It resolves relatively to 'process.cwd' by default, however
  you may want to use 'path.dirname(require.main.filename)'
> [debug] Cannot resolve 'npm3Package'
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/index.js
  Dynamic require may fail at run time, because the requested file
  is unknown at compilation time and not included into executable.
  Use a string literal as an argument for 'require', or leave it
  as is and specify the resolved file name in 'scripts' option.
> [debug] Cannot resolve 'npm2Package'
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/index.js
  Dynamic require may fail at run time, because the requested file
  is unknown at compilation time and not included into executable.
  Use a string literal as an argument for 'require', or leave it
  as is and specify the resolved file name in 'scripts' option.
> [debug] Cannot resolve 'topLevelPackage'
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/index.js
  Dynamic require may fail at run time, because the requested file
  is unknown at compilation time and not included into executable.
  Use a string literal as an argument for 'require', or leave it
  as is and specify the resolved file name in 'scripts' option.
> [debug] Bytecode of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/lib/verify-file.js is added to queue. It was required from /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/index.js
> [debug] Bytecode of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/package.json is added to queue. It was required from /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/index.js
> [debug] Stat info of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug is added to queue.
> [debug] Directory /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg is added to queue.
> [debug] Directory /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg is added to queue.
> [debug] Stat info of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/lib/verify-file.js is added to queue.
> [debug] Stat info of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/package.json is added to queue.
> [debug] Content of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/package.json is added to queue.
> [debug] Directory /Users/desmondkyeremeh/Projects/college-africa/tmp is added to queue.
> [debug] Stat info of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg is added to queue.
> [debug] Directory /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/lib is added to queue.
> [debug] Stat info of /Users/desmondkyeremeh/Projects/college-africa/tmp is added to queue.
> [debug] Directory /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer is added to queue.
> [debug] Stat info of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/lib is added to queue.
> [debug] Directory /Users/desmondkyeremeh/Projects/college-africa is added to queue.
> [debug] Stat info of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer is added to queue.
> [debug] Directory /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg is added to queue.
> [debug] Stat info of /Users/desmondkyeremeh/Projects/college-africa is added to queue.
> [debug] Directory /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules is added to queue.
> [debug] Directory /Users/desmondkyeremeh/Projects is added to queue.
> [debug] Stat info of /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules is added to queue.
> [debug] Stat info of /Users/desmondkyeremeh/Projects is added to queue.
> [debug] Directory /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug is added to queue.
> [debug] Directory /Users/desmondkyeremeh is added to queue.
> [debug] Stat info of /Users/desmondkyeremeh is added to queue.
> [debug] Directory /Users is added to queue.
> [debug] Stat info of /Users is added to queue.
> [debug] Directory / is added to queue.
> [debug] Stat info of / is added to queue.
> [debug] Deleting record file : /
> [debug] Deleting record file : /Users
> [debug] Deleting record file : /Users/desmondkyeremeh
> [debug] Deleting record file : /Users/desmondkyeremeh/Projects
> [debug] Deleting record file : /Users/desmondkyeremeh/Projects/college-africa
> [debug] The file was included as bytecode (no sources)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/index.js
> [debug] The file was included as bytecode (no sources)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/index.js
> [debug] The file was included as DISCLOSED code (with sources)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/package.json
> [debug] The file was included as DISCLOSED code (with sources)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/package.json
> [debug] The file was included as bytecode (no sources)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/index.js
> [debug] The file was included as bytecode (no sources)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/index.js
> [debug] files & folders deduped = 
  index.js
  node_modules
> [debug] The directory files list was included (2 items)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug
> [debug] The directory files list was included (2 items)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug
> [debug] The file was included as bytecode (no sources)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/lib/verify-file.js
> [debug] The file was included as bytecode (no sources)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/lib/verify-file.js
> [debug] files & folders deduped = 
  package.json
  index.js
  lib
> [debug] The directory files list was included (3 items)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg
> [debug] The directory files list was included (3 items)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg
> [debug] files & folders deduped = 
  pkg-debug
> [debug] The directory files list was included (1 item)
  /Users/desmondkyeremeh/Projects/college-africa/tmp
> [debug] The directory files list was included (1 item)
  /Users/desmondkyeremeh/Projects/college-africa/tmp
> [debug] files & folders deduped = 
  verify-file.js
> [debug] The directory files list was included (1 item)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/lib
> [debug] The directory files list was included (1 item)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer/ffmpeg/lib
> [debug] files & folders deduped = 
  ffmpeg
> [debug] The directory files list was included (1 item)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer
> [debug] The directory files list was included (1 item)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules/@ffmpeg-installer
> [debug] files & folders deduped = 
  @ffmpeg-installer
> [debug] The directory files list was included (1 item)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules
> [debug] The directory files list was included (1 item)
  /Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/node_modules
> [debug] Targets:
  [
  {
    "nodeRange": "node20",
    "platform": "macos",
    "arch": "arm64",
    "output": "/Users/desmondkyeremeh/Projects/college-africa/tmp/pkg-debug/build/app",
    "forceBuild": false,
    "fabricator": {
      "nodeRange": "node20",
      "platform": "macos",
      "arch": "arm64",
      "binaryPath": "/Users/desmondkyeremeh/.pkg-cache/v3.5/fetched-v20.17.0-macos-arm64-signed"
    },
    "binaryPath": "/Users/desmondkyeremeh/.pkg-cache/v3.5/fetched-v20.17.0-macos-arm64"
  }
]
robertsLando commented 2 months ago

When inside a packaged application you should ensure that the binaries you run are in filesystem and not on snapshot. What you can do is detect when you are running app in a packaged env and copy the binary to local filesystem before executing it:

const { promisify } = require('util');
const cp = require('child_process');
const fs = require('fs');
const { pipeline } = require('stream/promises');

let ffmpeg = require('@ffmpeg-installer/ffmpeg').path;

const start = async () => {

  if(process.pkg) {
    // copy ffmpeg to the current directory
    const file = fs.createWriteStream('ffmpeg');
    await pipeline(fs.createReadStream(ffmpeg), file);

    fs.chmodSync('ffmpeg', 0o755);
    console.log('ffmpeg copied to the current directory');
    ffmpeg = './ffmpeg';
  }

  cp.execSync(ffmpeg);
};

start();

In this way it works. Also ensure to run the command npx pkg . -t node18-linux-x64 -o build/app to build your application, without the . the package.json isn't picked up so assets are not loaded

dkyeremeh commented 2 months ago

Noted. Thank you for the support