atem-connection not playing well with webpack and electron.

[sensslen]

• I'm submitting a ... [ x ] bug report [ ] feature request [ ] question about the decisions made in the repository [ ] question about how to use this project

• Summary I am trying to use atem-connection in an electron application. Electron-Forge (the electron helper tool used) normally uses webpack in order to pack the application (and allow for usage of react.

When I use atem-connection on the main process, threadedClass fails to find the file atemSocketChild, as it searches the current directory.

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. StackOverflow, personal fork, etc.)

To reproduce the issue feel free to check out https://github.com/sensslen/Cgf.CameraControl.Main.Gui

[Julusian]

Could you try this version on npm 2.3.0-nightly-wip-update-threadedclass-20210313-231347-ec8d035.0. I think it should resolve your issues

[sensslen]

Thanks a lot for your answer. Unfortunately i does not. It seems that threadedClass still looks for a file instead of querying webpack for the file to load.

This is the Error I get:

(node:8800) UnhandledPromiseRejectionWarning: Error: Cannot find module 'D:\GIT\Private\Cgf.CameraControl.Main.Gui\.webpack\main\atemSocketChild'
Require stack:
- D:\GIT\Private\Cgf.CameraControl.Main.Gui\.webpack\main\index.js
- D:\GIT\Private\Cgf.CameraControl.Main.Gui\node_modules\electron\dist\resources\default_app.asar\main.js
-
    at Module._resolveFilename (internal/modules/cjs/loader.js:887:15)
    at Function.n._resolveFilename (electron/js2c/browser_init.js:261:1128)
    at Function.resolve (internal/modules/cjs/helpers.js:94:19)
    at D:\GIT\Private\Cgf.CameraControl.Main.Gui\.webpack\main\index.js:120749:40
    at new Promise (<anonymous>)
    at Object.threadedClass (D:\GIT\Private\Cgf.CameraControl.Main.Gui\.webpack\main\index.js:120640:12)
    at AtemSocket.<anonymous> (D:\GIT\Private\Cgf.CameraControl.Main.Gui\.webpack\main\index.js:17384:57)
    at Generator.next (<anonymous>)
    at D:\GIT\Private\Cgf.CameraControl.Main.Gui\.webpack\main\index.js:121484:71
    at new Promise (<anonymous>)

[Julusian]

oh sorry, I glossed over that you are using webpack.

Unfortunately I don't have any ideas on how to resolve this. I think the problem here is that it is trying to create a new worker thread, and the worker thread is failing to load its entrypoint. And even then the dynamic imports will probably have messed up the bundling so that the intended code will be missing. Webpack is not something I am comfortable with so do not know where to start

[sensslen]

No worries, neither do I have much knowledge about webpack.

would you mind telling me what purpose threadedclass serves so that I may try to spin a branch without it and thus make things work even with webpack?

[Julusian]

threadedclass is used to manage a worker thread. I am currently tempted to see if it is possible to swap out the library for https://github.com/andywer/threads.js, as that is more widely used and claims to work well with webpack.

The thread is necessary for handling the connection, to avoid the connection dropping when the node event loop is blocked by other tasks.

[sensslen]

Let me give this a try.

[sensslen]

Just to keep you up to date with my efforts. I'm currently trying to write a webpack plugin to handle threadedClass... This seems less intrusive than changing the used library.

[sensslen]

While workin on this issue, I found that webpack is not as easy to master as I initially thought. You mention the node event loop being blocked. Did you encounter this? Would it be feasable to spin a variant of atem-connection that runs everything on the event loop - I'd be willing to create and maintain this spin off...

[Julusian]

Yeah, I wouldnt want to dive into that side of webpack. Yes we found this very easy to hit. Even things as simple as parsing a chunk of json, or just while loops that dont yield.

If your code is using enough timers, or waiting on promises then it might be ok. But any synchronous operations that take more than 20ms are likely to cause issues.

[sensslen]

I found a way around this issue. There is the possibility to specify an external source for webpack (to tell it that this particular import should not be grabbed). This allows me to mark the atemSocketChild class as external. I the use WebpackCopyPlugin to copy the file over to the destination directory. With this trick I was able to get things to work properly. Thanks for the insights.

Maybe you want to add this to the documentation.

https://webpack.js.org/plugins/copy-webpack-plugin/ :

   new CopyPlugin({
        patterns: [{ from: '**/atemSocketChild*', to: '[name].[ext]' }],
    }),

(Note that the newest version does only support webpack 5)

https://v4.webpack.js.org/configuration/externals/ :

externals: /^(atemSocketChild)$/i,

[chrisspiegl]

I just ran into something that feels related to this. I am trying to build an application that has a Electron Menu Bar interface and using electron-builder I get this error message:

Worker Thread error Error: Cannot find module '/Users/*******/midi2atem/dist/mac/MIDI2ATEM.app/Contents/Resources/app.asar/node_modules/threadedclass/dist/child-process/threadedclass-worker.js'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:887:15)
    at Function.Module._load (internal/modules/cjs/loader.js:732:27)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
    at MessagePort.<anonymous> (internal/main/worker_thread.js:174:24)
    at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:354:41)
    at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26) {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Could this be related? And is there a possible workaround?

[sensslen]

This is definitively related. I was able to solve the issue by creating a second target for webpack that packages atemSocketChild class and it's dependencies. That way it works flawlessly. Please feel free to take https://github.com/sensslen/Cgf.CameraControl.Main.Gui (https://github.com/sensslen/Cgf.CameraControl.Main.Gui/blob/main/webpack.main.config.js) for inspiration.

[chrisspiegl]

@sensslen Thank you for sharing your config. At the moment I am probably just going to stick to unbuild versions… introducing webpack additionally was not on my plan for this moment.

Still hopeful that this lib could just support building out of the box.

[Julusian]

Im going to reopen this as a reminder to myself to look into what can be done here.

It's a tough one to handle, as webpack introduces some complexity (and I suspect that the config will need to in some way be aware of the need to do some extra work), and electron introduces some other complexity due to its bad support for worker_threads.

I have been tempted to try switching to a different thread helper library, which does itself have support for webpack, so it might make this easier to do neatly

[hrueger]

I have the same error as @chrisspiegl, however, I'm not using webpack or any other bundler. It's just plain js.

I've also made a test-repo. If you run npm start, it works but if you run npm run build and then run the dist/win-unpacked/electron-atem-connection-test.exe executable, you get the following errors in the terminal:

(node:19220) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 57)
Worker Thread error Error: Cannot find module 'A:\_Source\electron-atem-connection-test\dist\win-unpacked\resources\app.asar\node_modules\threadedclass\dist\child-process\threadedclass-worker.js'   
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:887:15)
    at Function.Module._load (internal/modules/cjs/loader.js:732:27)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
    at MessagePort.<anonymous> (internal/main/worker_thread.js:174:24)
    at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:354:41)
    at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26) {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Adding node_modules/threadedclass to the asarUnpack option of electron-builder does not help, because then the path would need to be [...]resources\app.asar.unpacked\node_modules\threadedclass[...].

Shoud I create an issue in the threadedclass repo? I think, if the user could configure the path to the threadedclass folder, it should work.

[Julusian]

@hrueger please try using version 3.0.0-nightly-latest-20210530-132658-16c3cdd.0. It is known that the 2.x versions do not like being packed in an asar, but the 3.0 versions are fine with it

[hrueger]

Thanks for the quick response, that worked 👍

[lolsborn]

I'm using the 3.0.0 nightly mentioned by @Julusian and am still running into issues with webpack.

(node:14441) UnhandledPromiseRejectionWarning: Error: Unable to resolve parent file path
    at eval (webpack:///./node_modules/threadedclass/dist/parent-process/threadedClass.js?:132:27)
    at new Promise (<anonymous>)
    at Object.threadedClass (webpack:///./node_modules/threadedclass/dist/parent-process/threadedClass.js?:29:12)
    at AtemSocket._createSocketProcess (webpack:///./node_modules/atem-connection/dist/lib/atemSocket.js?:81:53)
    at AtemSocket.connect (webpack:///./node_modules/atem-connection/dist/lib/atemSocket.js?:30:46)
    at Atem.connect (webpack:///./node_modules/atem-connection/dist/atem.js?:61:28)
    at eval (webpack:///./src/background.js?:24:8)
    at Module../src/background.js (/home/steven/Projects/galois/BESSPIN-Tool-Suite/besspin/cyberPhys/ui/hacker-kiosk/dist_electron/index.js:4393:1)
    at __webpack_require__ (/home/steven/Projects/galois/BESSPIN-Tool-Suite/besspin/cyberPhys/ui/hacker-kiosk/dist_electron/index.js:20:30)
    at eval (webpack:///multi_./src/background.js?:1:18)

[iosimliviu]

I'm also running into similar issues while running in the electron environment and trying to establish atem.connect('');.

Such as:

internal/worker.js:148 Uncaught (in promise) Error: The V8 platform used by this instance of Node does not support creating Workers
    at new Worker (internal/worker.js:148)
    at new WorkerThread (/Users/liviuiosim/Do…workerThreads.js:14)
    at Object.forkWorkerThread (/Users/liviuiosim/Do…workerThreads.js:52)
    at ThreadedClassManagerClassInternal._createFork (/Users/liviuiosim/Do…cess/manager.js:569)
    at ThreadedClassManagerClassInternal.findNextAvailableChild (/Users/liviuiosim/Do…cess/manager.js:126)
    at /Users/liviuiosim/Do…hreadedClass.js:144
    at new Promise (<anonymous>)
    at Object.threadedClass (/Users/liviuiosim/Do…threadedClass.js:29)
    at AtemSocket.<anonymous> (/Users/liviuiosim/Do…ib/atemSocket.js:90)
    at Generator.next (<anonymous>)

Have there been any updates on this topic?

[sensslen]

After I was able to fix the webpack stuff I also encountered this issue - just to find out that electron‘s worker thread implementation is kind of broken. At least that‘s what I read. Therefore I moved away from electron.