pinojs / thread-stream

A streaming way to send data to a Node.js Worker Thread
MIT License
229 stars 23 forks source link

Import error when using worker with pkg #143

Closed nagyszabi closed 5 months ago

nagyszabi commented 6 months ago

Hi there!

When using pkg (more specifically yao-pkg) the worker is unable to load files. This is happening on node v20 while using CommonJS modules throughout the whole project, and it's throwing the following error:

TypeError [ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING]: A dynamic import callback was not specified.
    at importModuleDynamicallyCallback (node:internal/modules/esm/utils:213:9)
    at eval (eval at <anonymous> (C:\snapshot\backend\node_modules\real-require\src\index.js:4:20), <anonymous>:3:1)
    at start (C:\snapshot\backend\node_modules\thread-stream\lib\worker.js:33:23)
    at Object.<anonymous> (C:\snapshot\backend\node_modules\thread-stream\lib\worker.js:91:1)
    at Module._compile (pkg/prelude/bootstrap.js:1930:22)
    at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)
    at Module.load (node:internal/modules/cjs/loader:1207:32)
    at Module._load (node:internal/modules/cjs/loader:1023:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:135:12)
    at MessagePort.<anonymous> (node:internal/main/worker_thread:186:26)
Emitted 'error' event on ThreadStream instance at:
    at Immediate.<anonymous> (C:\snapshot\backend\node_modules\thread-stream\index.js:366:12)
    at process.processImmediate (node:internal/timers:478:21)

Node.js v20.11.1

Which is thrown because it tries to use import, fails, and then none of the cases match, so it throws the error again.

I saw that some similar bug has happened before, and that using require fixed the issue, so I tried just that, and doing:

} else if (error.code === undefined || error.code === 'ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING') { // adding error here
  // When bundled with pkg, an undefined error is thrown when called with realImport
  worker = realRequire(decodeURIComponent(filename.replace(process.platform === 'win32' ? 'file:///' : 'file://', '')))
}

Actually fixes the problem.

I do not know how could I write a test for this case, or if you could maybe give some insight on why the error is thrown, because I was unable to find any reason on why this would happen.

mcollina commented 6 months ago

I have no clue, but a PR would be highly welcomed.